Transformers and large language models in healthcare: A review

Abstract

With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning architecture initially developed to solve general-purpose Natural Language Processing (NLP) tasks and has subsequently been adapted in many fields, including healthcare. In this survey paper, we provide an overview of how this architecture has been adopted to analyze various forms of healthcare data, including clinical NLP, medical imaging, structured Electronic Health Records (EHR), social media, bio-physiological signals, biomolecular sequences. Furthermore, which have also include the articles that used the transformer architecture for generating surgical instructions and predicting adverse outcomes after surgeries under the umbrella of critical care. Under diverse settings, these models have been used for clinical diagnosis, report generation, data reconstruction, and drug/protein synthesis. Finally, we also discuss the benefits and limitations of using transformers in healthcare and examine issues such as computational cost, model interpretability, fairness, alignment with human values, ethical implications, and environmental impact.

1. Introduction

The last decade has seen an explosion in data generated by healthcare practices. Currently, healthcare data accounts for 30% of the global data ecosystem and is expected to grow in the coming years [1]. The increasing availability of digital patient data has enabled the development of machine learning algorithms to support diagnosis, prognosis, and clinical decision-making.

Transformer [2] is a type of Deep Neural Network (DNN) introduced in 2017 for sequence modeling problems, especially in the Natural Language Processing (NLP) domain [3]. Before the introduction of the Transformer [2], the most popular sequential deep learning architectures, such as recurrent neural networks (RNNs) [4] and their variants, worked in a serial fashion which precluded parallelization during training, therefore substantially increasing the training time. In contrast, Transformer employs parallelizable scaled dot-product attention mechanism. This unique attention mechanism allows for large-scale pretraining. Additionally, self-supervised pretraining on large unlabeled datasets using approaches such as input masking enabled transformers to be trained without costly annotations.

Although originally designed for the NLP [3] domain, Transformers have witnessed adaptations in various domains such as computer vision [5,6], remote sensing [7], time series [8], speech processing [9] and multimodal learning [10]. Consequently, modality-specific surveys emerged, focusing on medical imaging [11–13] and biomedical language models [14]. However, a comprehensive review of all healthcare-oriented literature that has employed the Transformer architecture has not been undertaken. This paper aims to provide a comprehensive review of Transformer models utilized across multiple healthcare data modalities while focusing on notable architectural changes undergone by the original Transformer model through this process of evolution. This is critical and timely because the transformer architecture is being rapidly incorporated into almost every healthcare domain; it is critical to understand common patterns and features in this adoption. Furthermore, we discuss pretraining strategies designed to manage the lack of robust and/or annotated healthcare datasets. The rest of the paper is organized as follows: Section 2 discusses our search strategy; Section 3 describes the architecture of the original transformer; Section 4 describes the two most commonly used Transformer variants: the Bidirectional Encoder Representations from Transformers (BERT) and the Vision Transformer (ViT). Section 5 describes advancements in large language models (LLM), and Sections 6 through 12 provide a review of Transformers in healthcare. Finally, Section 13 discusses limitations, interpretability, environmental impact, computational costs, bias, and fairness. This review summarizes the use of transformer-based deep learning models in healthcare and provides a critical analysis of the inherent deficiencies of these models and discusses possible future directions for this field.

2. Search strategy and selection criteria

We used Google Scholar and PubMed search engines to search for studies. Since Vaswani et al.’s initial Transformer network [2] was published in 2017, we limited our search to studies published after 2017. We also limited our search to studies published before March 2023 to complete this review. The extraction process and exclusion criteria is shown in Fig. 1. The search was divided into six categories: clinical NLP, EHR, social media, medical imaging, biomolecules, and bio-physical signals.

Fig. 1.

Flow diagram depicting the process for selecting relevant studies for inclusion and exclusion.

For each category, we used the terms “health” or “medical” or “clinical” to focus the search on the healthcare domain. Finally, each category used a precise set of keywords unique to that domain. The keywords were combined with logical operators such as “AND” and “OR” to enhance search fidelity. A detailed list of search queries can be found in Table 1. We used Harzing’s Publish or Perish [15] to retrieve studies and Covidence [16] to select relevant studies. For most medical domains an article that was preliminarily short-listed was reviewed by one reviewer. Six reviewers worked independently on studies pertaining to different domains. For topics which had a significantly larger number of papers (for e.g., clinical NLP and medical imaging) three reviewers worked together to analyze relevant articles, and only those articles were retained which were deemed relevant by all three reviewers.

Table 1.

Search queries used to extract relevant studies for each topic.

Topic	Search query
Clinical NLP	(“coreference” OR (“semantic textual similarity” OR STS) OR (“named entity recognition” OR NER) OR “relation extraction” OR “natural language inference” OR “question answering” OR “entity normalization”) AND (BERT OR Transformer) AND (“clinical” OR “medical” OR “biomedical” OR “EHR”) from 2017
Structured EHR	(Transformer OR BERT) AND (“deep learning” OR “machine learning”) AND (EHR OR “electronic health records”) from 2017
Medical Imaging	(Segmentation OR registration OR “image captioning” OR “report generation” OR “visual question answering” OR “image synthesis” OR “classification” OR “reconstruction”) AND (“Transformer” OR “vision transformer”) AND (“clinical” OR “medical” OR “biomedical” OR “EHR”) from 2017
Critical Care	(Transformer) AND (“deep learning” OR “machine learning”) AND (“critical care” OR “surgery” OR “surgical”) from 2017
Social Media	(Transformer OR BERT) AND (“deep learning” OR “machine learning”) AND (“social media” OR “crowdsource” OR “crowdsourcing” OR “twitter” OR “tweet”) from 2017
Bio-physical Signals	(Transformer OR BERT) AND (“deep learning” OR “machine learning”) AND (“medical” OR “health” OR “clinical” OR “biomedical”) AND (“signal” OR “ECG” OR “EMG” OR “EEG” OR “human activity” OR “HAR”) from 2017
Biomolecular Sequences	(Transformer OR BERT) AND (“deep learning” OR “machine learning”) AND (DNA OR RNA OR gene OR genome OR genomic OR transcriptomic OR protein OR proteomic OR metabolite OR metabolism OR metabolomic OR chromosome OR receptor OR mitochondria OR splicing) from 2017

We identified the top keywords from articles included in this report to provide an overview of key concepts, data modalities, and tasks. The word cloud in Fig. 2 shows the 50 most common keywords across articles, with a larger font representing more papers; while Fig. 3 shows data modalities and the corresponding tasks.

Fig. 2.

Word cloud depiction of keywords used in the surveyed literature. Abbreviations. BERT; Bidirectional Encoder Representations from Transformers, CNN; Convolutional Neural Networks, EHR; Electronic Health Records, MRI; Magnetic Resonance Imaging, NER; Named Entity Recognition, NLP; Natural Language Processing, STS; Semantic Textual Similarity.

Fig. 3.

Major healthcare data modalities and corresponding tasks. Abbreviations: EEG; Electroencephalography, ECG; Electrocardiogram, NER; Named Entity Recognition, RE; Relation Extraction, STS; Semantic Textual Similarity, ICD; International Classification of Diseases, EHR; Electronic Health Record.

3. Background

Transformers are multilayered neural networks formed by stacking either multiple encoder and/or decoder blocks which utilize the attention mechanism, as explained in the following section.

3.1. Attention

The attention mechanism computes similarity between individual input tokens, such as vectors of word embeddings. In a basic Transformer architecture, each input embedding generally can take three roles: (1) Query $Q$ which is the current focus of the attention mechanism and is being compared to all other input tokens, (2) Key $K$ is the input token being compared to the query, and (3) Value $V$ is a value used to compute the output of attention. The attention function can be considered a mapping between a query and a set of key-value pairs to produce an output [2].

We will represent the input $X\in R^{n\times d}$ as a sequence of $n$ tokens with an embedding dimension of $d$. The input sequence $X$ is linearly transformed into query $Q$, key $K$, and value $V$ using Eqs. (1), (2), and (3), respectively.

$$Q=X•W_q$$ (1)

$$K=X•W_k$$ (2)

$$V=X•W_v$$ (3)

where ${\mathrm{W}}_q$, ${\mathrm{W}}_k$, and ${\mathrm{W}}_v$ are the weight matrices to obtain $Q$, $K$, and $V$ matrices. The $Q$, $K$, and $V$ are then used to compute the scaled dot product attention as shown in Equation.

$$Attention(Q,K,V)=softmax\left(\alpha Q•K^T\right)•V$$ (4)

In Eq. (4), the scaled dot product operation is performed between the query and key matrices, followed by a softmax function. Here, the scale factor $\alpha$ is used to mitigate the vanishing gradient problem or numerical instability. It is typically chosen to be $1/\left(\surd d_k\right)$, where $d_k$ is the key dimension.

3.2. Attention mechanisms

Transformer models primarily use three types of attention: self-attention, masked self-attention, and cross-attention.

3.2.1. Self-attention

Self-attention is when attention is computed between tokens in the same sequence. The self-attention block is found in the Transformer encoder. The dimensions of query, key, and value are the same in self-attention, i.e., $d_k=d_q=d_v$.

3.2.2. Masked self-attention

In sequence prediction problems, such as machine translation, the context of previous tokens in a sequence is used to predict the subsequent output. A mask is typically employed to prevent the model from attending to subsequent tokens in a sequence. The mask $M$ is a square upper triangular matrix with dimension $n$, where $n$ is the number of tokens in the input sequence. The mask is applied to the scaled dot product of the query and key via element-wise addition, as in Eq. (6)

$$MaskedAttention(Q,K,V)=softmax\left(\frac{Q•K^T}{\sqrt{d_k}}+M\right)V$$ (6)

3.2.3. Cross-attention

Cross-attention is attention computed between tokens of one sequence with tokens of another sequence. In Transformer, the input and output sequences interact through cross-attention in the decoder module. The cross-attention module receives its queries from the previous masked self-attention layer of the decoder and its keys and values from the last encoder. Queries correspond to the desired output sequence, while the keys and values are generated based on the input sequence in the encoder.

3.2.4. Multi-head attention

It has been shown that compared to a single attention computation, multiple attention operations can improve the model’s performance by capturing different similarity relationships in the sequence [2]. The attention blocks in both the encoder and decoder are computed with $h$ attention heads, as shown in Fig. 4. The original Transformer model employed $h=8$ attention heads. Every attention head has three learnable weight matrices: $W_q^i$, $W_k^i$, and $W_v^i$, where $i$ represents a particular attention head. The attention outputs from multiple heads (denoted by $\mathscr{H}$) are then concatenated and linearly transformed to the model dimension with a parameter matrix $W_{\text{o}}$.

Fig. 4.

Multi-head attention mechanism. In the encoder and decoder, multiple attention heads are stacked together and their outputs are concatenated.

3.3. Position-wise feed-forward network

The output of the attention modules is passed to a two-layered feedforward network (FFN). The FFN performs an independent position-wise linear transformation on each token of the sequence. Parameters of this network are shared across all positions of the sequence.

Let $\mathscr{H}$ be the output of the multi-head attention block and $d$ be the model dimension. The first linear layer transforms $\mathscr{H}$ from dimension $d$ to an intermediate dimension $d_f$, also referred to as the feedforward dimension. The second linear layer transforms the output of the first linear layer from $d_f$ to the original model dimension $d$. The FFN is given by Eq. (9).

$$ℱ\left(H\right)=ReLU\left(\mathscr{H}•w_1+b_1\right)•w_2+b_2$$ (9)

The intermediate dimension $d_f$ is usually set to a value larger than $d$.

3.4. Residual connections and layer normalization

Residual connections [17] allow gradients to skip non-linear activation functions, followed by layer normalization. Layer normalization scales the values of all hidden layers to a similar range to avoid exploding or diminishing values obtained through a chain of multiplication operations.

3.5. Positional encodings

Because the self-attention module attends to all tokens of a sequence in parallel, it intrinsically neglects the order of tokens in the sequence. This necessitates using a positional encoding ($PE$) vector that denotes the unique position of each token. Transformers use a combination of sine and cosine functions of different frequencies to create PE vectors shown in Eq. (10). PE vectors are added to the embeddings of each input token; therefore the PE dimension is chosen to be the same as the embedding dimension. Since sine and cosine functions have values in the range [−1, 1], the values of the positional encoding matrix are constrained to a normalized range. This technique enables Transformers to capture the relationship between items that are both close and far from one another in a sequence.

$$\begin{gathered} P{E}_{(pos,i)}=\left\{\begin{array}{l}sin\left(pos•{\omega }_k\right),if\\ cos\left(pos•{\omega }_k\right),ifi=2k+1\end{array}\right.i=2k \\ {\omega }_k=\frac{1}{{10000}^{\raisebox{1ex}{$2k$}\!\left/ \!\raisebox{-1ex}{$d$}\right.}},k=1,2,\dots ,\frac{d}{2} \end{gathered}$$ (10)

3.6. Assembling a transformer

Transformer consists of an encoder and a decoder network. The encoder consists of identical encoder blocks stacked upon each other, each consisting of a self-attention and an FFN layer. The decoder consists of stacked identical decoder blocks, each consisting of a masked self-attention layer, cross-attention layer, and FFN layer. The encoder transforms an input sequence into encoded representations, while the decoder operates upon these representations.

The original Transformer architecture (Vaswani et al., 2017) [2], shown in Fig. 5, has six identical stacked encoders and six identical stacked decoder blocks. Each encoder block comprise of multi-head self-attention followed by FFN. Every decoder block consists of multi-head masked self-attention, multi-head cross-attention, and FFN arranged sequentially. The cross-attention layers attend to queries from the previous masked attention layers, whereas keys and values are obtained from the output of the final encoder block. The output of the last encoder is used to obtain the keys and values to compute the multi-head cross attention in all the decoder layers.

Fig. 5.

Schematic of the transformer architecture [2,18].

3.7. Computational complexity of transformer attention

The self-attention mechanism of Transformer can attend to variable-length input size but has $O\left(n^2\bullet d\right)$ time complexity where $n$ and $d$ are the input sequence length and the model dimension. For long input sequences, this attention computation becomes computationally expensive. Many Transformer variants try to reduce the computational complexity via different approaches [19].

3.8. Transformer model usage

In general, Transformer architectures can be divided into three categories.

Encoder-Decoder: consists of multiple encoder and decoder blocks and is typically used in sequence-to-sequence modeling tasks, such as machine translation.

Encoder only: Only the encoder blocks are used to model the input sequence. The output of the encoder is a contextual representation of the input sequence. This type of architecture is used for classification or label prediction problems (most models in this review).

Decoder only: Only decoder blocks are used. This architecture is used for sequence generation, image captioning, and language modeling tasks.

4. Mainstream transformer-based architectures

In this section, we will discuss the two prominent transformer-based architectures with significant impact on NLP and computer vision.

4.1. Bidirectional encoder representations from transformers (BERT)

BERT [20], is an encoder-only Transformer architecture that can produce rich contextualized word and sentence embeddings for NLP. Unlike traditional language models, which read text input sequentially (left-to-right or right-to-left), the Transformer encoder in BERT reads the entire sequence of words at once, thereby learning a richer representation of context and information flow in a sentence. The BERT architecture uses self-supervised pretraining steps, namely Masked Language Modeling (MLM), to create context-sensitive word embeddings, and Next Sentence Prediction (NSP) to model sequential association between sentences. MLM masks a fraction of the input tokens and aims to predict them based on their context. This helps to disentangle ambiguity in the text by using surrounding text to establish context. In NSP, a combination of two sentences is fed to the Transformer encoder. In 50% of cases, the second sentence is the next sentence in the original text, while in the remaining 50% of cases, the second sentence is randomly selected. The encoder learns to distinguish scenarios where the sentences are logically linked. When training the BERT model, MLM and NSP are trained together to minimize the combined loss function of the two strategies. BERT can be used for various language tasks, such as sentence classification, Question Answering (QA), and Named Entity Recognition (NER) with finetuning and minor modifications to the original architecture.

4.2. Vision Transformer (ViT)

ViT is a pure Transformer architecture without convolutional layers and was proposed for image classification tasks [1]. Like BERT, ViT is also an encoder-only Transformer model. Transformers cannot directly process spatial data such as images; therefore, data must be converted to a sequence. ViT splits an image into fixed-size patches, generally 16 × 16 or 32 × 32 flattened, before they are provided as an input to the transformer model. The flattened patches are placed in a sequence, then transformed into a low-dimensional linear embedding. Like the original Transformer, PEs are added to the linear embeddings to inject information about each patch’s relative location in the image, where 1D, 2D, and learnable positional embeddings can be used. An extra learnable class embedding is added at the start of the sequence, used for downstream classification tasks. During fine-tuning, a classification head comprised of a single hidden layer network is attached to this class embedding.

Transformer models by design do not possess the inductive biases of CNNs, such as limited receptive field and translational invariance (ability to detect or recognize an object regardless of its location in an image). In CNNs, the receptive field increases linearly with the depth of the model. While the Transformer lacks the inductive biases of the CNN, they are permutation invariant (not dependent on the order of elements in a sequence), and the shallow layers of the model can attend to the entire image.

5. Large language models (LLMS)

Foundation models are large-scale AI systems trained on vast amounts of data to be adapted for a wide range of downstream tasks [21]. LLMs colloquially refer to a class of foundation models with billions of parameters trained on language corpora with billions of words to generate human-like language and solve different NLP tasks. Most LLMs use the Transformer architecture, the current default architecture for processing sequential data as of 2023. The success of LLMs comes from the self-supervised pre-training paradigm, which takes advantage of large free text data without annotation. This pre-training technique enabled LLMs to generate coherent and realistic language, making them useful for various applications such as text completion, dialogue generation, and content generation. BERT style LLMs( Encoder-Decoder or Encoder only) are pretrained using masked language modeling while GPT style (decoder only) models are pretrained by generating next work in a sequence. Generative AI models trained to generate text and question answering tasks are autoregressive decoder-only language models. Examples of autoregressive decoder-only language models include PaLM [22], GPT-3 [23], Chinchilla, LLaMA [24], PaLM2 [25] used in BARD chatbot, and GPT-4 [26]. These models are trained on billions of tokens obtained from datasets such as Common Crawl, WebText2, Books1, Books2, Wikipedia, Stack Exchange, PubMed, ArXiv, Github, Gutenberg, and many more. Some of the domain-specific LLMs include Galactica [27], trained on curated human scientific knowledge corpora, BloombergGPT [28], trained on proprietary financial data, and CodeX [29] for code generation. A timeline of popular LLMs is displayed in Fig. 6.

Fig. 6.

The timeline of popular large language models developed over the years (2018–2023).

The number of parameters in LLMs and the size of their training data has increased rapidly, reaching up to trillions of tokens [24]. The capabilities of LLMs appear to be a function of the amount of data, parameters, and computation resources rather than architectural design advancements [30]. The scaled-up language models develop abilities beyond the trained outcomes called ‘emergent abilities,’ which are not designed but discovered after deployment [31]. For example, GPT-3 showed few-shot prompting ability; when provided few input-outputs for a natural language task, the model can perform the task on unseen samples without further training or gradient updates to the parameters [23]. Parameter-efficient models such as Stanford Alpaca [32] and efficient finetuning approaches of Quantized LLMs such as QLoRA [32] have been introduced to address situations where computational resources are limited. Despite the exceptional ability of LLMs to generate realistic text, they can also generate false information, toxic language, and racial stereotypes [33,34].

In the medical domain, Agrawal et al. [35] demonstrated that LLMs can be few-shot clinical information extractors without further training on the clinical data. They used InstructGPT [36] for this task, significantly outperforming existing zero-shot and few-shot baselines. In Radiology, Jeblick et al. [37] performed an exploratory case study to evaluate ChatGPT’s ability to simplify radiology reports. Expert human radiologists considered the simplified reports complete, factual, and devoid of harmful text that could misguide the patient. However, instances of missing key findings and incorrect statements were observed. The PMC-LLaMA [38] model, fine-tuned on 4.8 million biomedical papers obtained from PubMed Central, demonstrated a better understanding of biomedical domain-specific concepts than the original LLaMa when evaluated on biomedical QA benchmarks. GatorTron [39], a large clinical language model with 8.9 billion parameters trained on over 90 billion words of clinical text, was applied to clinical NLP tasks such as clinical concept extraction. Luo et al. [40] proposed BioGPT, a biomedical domain specific generative model pretrained on PubMed abstract corpus to generate fluent biomedical term descriptions.

Singhal et al. [41] evaluated the 540 billion parameters PaLM [22] and its variant FLAN-PaLM [42] on the benchmark dataset MultiMedQA. This benchmark dataset combines multiple QA datasets, including medical exams, consumer queries, and research. The authors also introduced Med-PaLM, a parameter-efficient model that used prompt instruction tuning to fix the critical Flan-PaLM gaps observed upon human evaluation. In subsequent work, Singhal et al. proposed Med-PaLM2 [43] to bridge the gap between the model’s answers to that of clinicians. The model combines improvements that come with PaLM2 [25], a novel ensemble refinement prompting strategy, and domain-specific model fine-tuning. Scaled-up models such as ChatGPT, PaLM, PALM2, and GPT-4 have been shown to answer medical questions and successfully pass or achieve near-passing scores on medical licensing examinations [41,44–47]. These existing large medical foundation models trained on broad biomedical domain corpora such as PubMed are tested on tasks with minimal significance to the health systems [48]. The impressive advancements of foundation models have not yet permeated into medical AI. These early approaches are limited by a lack of large, diverse medical datasets, the complex nature of medical data, federal patient data privacy regulations, and the recency of the general-purpose foundation models [49]. Transformers in NLP

6. Transformers in clinical NLP

6.1. Clinical word embeddings

Word embeddings map variable-length words to a fixed-length vector while preserving syntactic and semantic information. Word embeddings are a standard representation used in NLP. Traditional word embedding techniques such as word2vec [50] or GLoVe [51] learn an aggregated representation of all contexts associated with a word. Previously contextual word embedding based on models such as ELMo [52], BERT [20], and ULMFiT [53] achieved state of the art (SOTA) performance on NLP tasks. However, these embeddings cannot be adapted directly to clinical or biomedical text due to differences in the linguistic domain corpora. Lee et al. [50] introduced BioBERT, a pre-trained language model in the biomedical domain, to overcome this difficulty. BioBERT is initialized with BERT weights and is pre-trained on PubMed Central full-text articles and abstracts as shown in Fig. 7. This pre-trained model is fine-tuned on three popular biomedical NLP tasks: Named Entity Recognition (NER), Relation Extraction (RE), and QA. BioBERT has outperformed previous models on biomedical text mining tasks with minimal task-specific modification.

Fig. 7.

BioBERT pre-training and finetuning overview. Source: Image adapted from [54] without modifications.

Further specialization of BERT and BioBERT via pre-training on specific EHR databases has proven promising. Alsentzer et al. [55] pre-trained BERT and BioBERT on 2 million clinical notes from the MIMIC-III database [56] to obtain clinical BERT and Bio+Clinical BERT. Si et al. [57] explored various embedding methods such as word2vec [50], GloVe [51], fastText [58], ELMo [52], and BERT [20] on clinical concept extraction tasks to demonstrate the generalizability of these traditional embedding methods. When pre-trained on a clinical domain-specific corpus [56], all the embeddings yielded increased performance. Huang et al. [59] pretrained BERT [20] on clinical notes from the MIMIC-III dataset [56] to develop ClinicalBERT. ClinicalBERT achieved higher Pearson correlation scores than word2vec [50] and fastText [58]. All these models were pre-trained on clinical domain corpora and have outperformed models pre-trained on general or biomedical domain corpora in clinical NLP tasks.

6.2. Transformers for clinical information extraction (IE)

EHRs contain a wealth of patient information stored in structured and unstructured formats, including detailed clinical notes used for documentation. Parsing through this data is difficult due to the unstructured nature of the free text entries recorded by clinical staff in the EHR. Clinical IE consists of sub-tasks such as NER, coreference resolution (CR), QA, semantic textual similarity (STS), relation extraction (RE), and entity normalization (EN). The success of Transformers inspired researchers to adapt Transformer-based architectures for clinical IE (Table 2).

Table 2.

Transformers in clinical NLP.

Reference	Title	Tasks	Datasets	Architecture
Lee et al. [54]	BioBERT: a pre-trained biomedical language representation model for biomedical text mining	NER, RE, QA	NCBI Disease [60], I2b2 2010 [61], BC5CDR [62], BC4CHEMD [63], BC2GM [64], JNLPBA [65], LINNAEUS [66], Species-800 [67], GAD [68], EU-ADR [69], CHEMPROT [70], BioASQ [71]	BERT[20]
Alsentzer et al. [55]	Publicly available clinical BERT embeddings	NLI, NER, de identification, concept extraction, entity extraction	MIMIC-III [56], i2b2 2010 [61], i2b2 2012 [72,73], MedNLI [74], i2b2 2006 [75], i2b2 2014 [76,77]	BERT [20]
Si et al. [57]	Enhancing clinical concept extraction with contextual embeddings	Concept extraction	i2b2 2010 [61], i2b2 2012 [72], i2b2 2014 [76], ShARe/CLEF [78,79], SemEval [80–82], MIMIC-III [56]	BERT [20]
Peng et al. [83]	BlueBERT: Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets	SS, NER, RE, DC, Inference	MEDSTS [84], BIOSSES [85], BC5CDR [62], ShARe/CLEF [78], DDI [86], CHEMPROT [70], i2b2 2010 [61], HoC [87], MedNLI [74]	BERT[20]
Gu et al. [88]	Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing	NER, RE, SS, DC, QA	NCBI Disease [60], BC5CDR [62], BC2GM [64], JNLPBA [89], CHEMPROT [70], DDI [86], GAD [68], BIOSSES [85], HoC [87], PubMedQA [90] BioASQ [71,91]	PubMedBERT
Huang et al. [59]	ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission	Patient readmission prediction	MIMIC-III [56]	BERT [20]
Yang et al. [92]	Clinical concept extraction using transformers	Concept extraction	MIMIC-III [56], i2b2 2010 [61], i2b2 2012 [72,73], n2c2 2018 [93,94]	BERT [20], RoBERTa [95], ALBERT [96], ELECTRA [97]
Wei et al. [98]	Relation Extraction from Clinical Narratives Using Pre-trained Language Models	RE	n2c2 2018 [93,94]. i2b2 2010 [61]	BERT [20]
Mayer et al. [99]	Transformer-Based Argument Mining for Healthcare Applications	Argument component detection, RE	MEDLINE	BERT [20], BioBERT [54], SciBERT [100], RoBERTA [95]
Huang et al. [101]	Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation	Prognosis prediction	MIMIC III [56]	XLNet [102], BERT [20], ClinicalBERT [59],
Yu et al. [103]	BioBERT based named entity recognition in electronic medical record	NER	I2b2 2010 [61]	BioBERT[54]
Alimova et al. [104]	Multiple features for clinical relation extraction: A machine learning approach	RE	n2c2 2018 [93,94], MADE 2018 [105]	BERT [20], BioBERT [54], ClinicalBERT [59]
Jin et al. [90]	PubMedQA: A dataset for biomedical research question answering	QA	PubMedQA [90]	BioBERT [54]
Yoon et al. [106]	Pre-trained language model for biomedical question answering	QA	SQuAD [107,108], BioASQ [71,91]	BioBERT [54]
Ji et al. [109]	BERT-based ranking for biomedical entity normalization	EN	ShARe/CLEF [110], NCBI [60], TAC2017ADR [111]	BERT [20], BioBERT [54], ClinicalBERT [55]
Yang et al. [112]	Measurement of Semantic Textual Similarity in Clinical Texts: Comparison of Transformer-Based Models	STS	2019 n2c2/ Open Health NLP [113]	BERT [20], XLnet[102], RoBERTa [95]

NER: Named Entity Recognition; SS: Sentence Similarity; RE: Relation Extraction; DC: Document Classification; NLI: Natural Language Inference; QA: Question Answering, EN: Entity Normalization; STS: Semantic Textual Similarity

6.2.1. Named entity recognition

Clinical named entity recognition (CNER) aims to identify entities, concepts, and events such as diseases, drugs, treatments, medical conditions, and symptoms from clinical narratives. CNER is challenging as clinicians often use acronyms and abbreviations to describe complex clinical terms without using standardized clinical ontology. Earlier approaches used the BERT model to generate clinical textual embeddings, which were further used to train other deep learning models, such as BiLSTM and conditional random fields [114–116]. Later, for biomedical and clinical domains, domain-specific BERT-based models such as BioBERT [54] and clinical BERT by Alsentzer et al. [55] established baselines on CNER datasets. BERT-based models have been applied to CNER tasks in different languages, such as Chinese [117,118], Korean [119], Italian [120], Spanish [121], and Arabic [122].

The clinical de-identification task, which removes protected health information, was also approached as a NER problem by pretrained BERT-based models, such as clinical-BERT [55] and UMLS-BERT [123]. These models were applied to i2b2–2006 [75] and i2b2–2014 [77] de-identification tasks. Garcia et al. [124] and Mao et al. [125] used BERT on the MEDDOCAN [126] Spanish de-identification corpus.

The clinical concept extraction task predicts a concept’s start and end positions in a document. BIO tags are commonly used, where “B”, “I”, and “O” refer to the beginning, inside, and outside of a concept. Yang et al. [92] developed an open-source Transformers package with four transformer-based models, BERT [20], ALBERT [96], RoBERTa [95], and ELECTRA [97], pretrained on MIMIC-III dataset for clinical concept extraction. Peng et al. [83] used transfer learning to fine-tune BERT [20] for concept extraction on BC5CDR [62] and ShARe/CLEF [110] datasets. Khan et al. [127] proposed MT-BioNER, a transformer-based model for intent classification and slot tagging. The authors combined BERT encoder layers with task-specific layers to train their model on NCBI-disease [128], BC5CDR [62], and JNLPBA [89] datasets.

6.2.2. Clinical coreference resolution (CR)

The CR task aims to identify all mentions of the same entity in a text. Trieu et al. [129] performed CR in full-text articles as part of the CRAFT 2019 shared task [130]. The authors employed a span-based end-to-model proposed by Lee et al.[131] and replaced the LSTM layers with BERT. Their results on the CRAFT coreference resolution task indicate the effectiveness of BERT in capturing long-distance coreferences in large documents. Steinkamp et al. [132] used BERT [20] to perform CR for symptom extraction on the i2b2 2009 Medication Challenge [133] and MIMIC-III datasets [56], showing better performance compared to recurrent models.

6.2.3. Clinical relationship extraction (CRE)

CRE is categorized into concept relationship and temporal relationship extraction. Concept relationship extraction identifies the relationship between two concepts (e.g., drug and dosage), whereas temporal relationship extraction evaluates the relationship between clinical events occurring at different times. Peng et al. [83] approached the CRE task as a sentence classification problem by replacing named entity mentions of interest with pre-defined tags using BERT [20] on DDI [86], ChemProt [70], and i2b2 2010 [61] datasets. Wei et al. [98] fine-tuned BERT outperformed SOTA RE models on clinical RE tasks using n2c2–2018 [94] and i2b2–2010 [61] datasets. Zhang et al. [114] pretrained the BERT model on Chinese clinical text and fine-tuned on the breast cancer dataset to classify the relationship between clinical concepts and corresponding attributes for breast cancer. Using BERT, Xue et al. [129] used an integrated joint learning approach for NER and CRE in coronary angiography Chinese clinical text. Lai et al. [134] proposed BERT-GT, which combines BERT with Graph Transformer by integrating the neighbor attention mechanism into BERT. BERT-GT was used for cross-sentence RE on the N-ary [135] and BioCreative CDR [136] datasets. Lin et al. [137] developed a pre-trained BERT model on the MIMIC-III dataset and BioBERT [54] models for temporal RE on the THYME [138] corpus. Their BioBERT model with sentence agnostic 60-token window approach was used for the CONTAINS temporal relation extraction task on the colon cancer test set.

6.2.4. Question answering (QA)

The QA ability of a model can serve as an indicator of its ability to learn the medical text. Jin et al. [90] introduced the PubMedQA dataset for biomedical research question answering, and fine-tuned BioBERT model to establish a baseline on the dataset. Yoon et al. [106] pretrained the BioBERT model on SQuAD [107,108] datasets and fine-tuned it for the BioASQ [71,91] biomedical QA challenge. This model achieved SOTA performance on factoid, list, and yes/no type questions of the BioASQ dataset. He et al. [139] proposed a procedure for consumer health question answering and medical language inference tasks using models such as BERT[20], BioBERT[54], SciBERT[100], ClinicalBERT [55], BlueBERT[83], and ALBERT[96]. Schmidt et al. [140] developed a QA-BERT model for question answering using the PICO (Population, Intervention, Comparator, and Outcome) framework. The PICO element dataset [141] was combined with SQuAD datasets [107,108] to increase the generalizability and flexibility of the model on all types of questions. The proposed QA-BERT performed better than LSTM and BERT baselines [140].

6.2.5. Biomedical entity normalization (BEN)

BEN aims to link mentions of an entity in a clinical document (e.g., EHR) to their corresponding concepts in a knowledge base [142]. Ji et al. [109] fine-tuned pre-trained models such as BERT [20], BioBERT [54], ClinicalBERT [55] on three different datasets ShARe/CLEF [110], NCBI [60], TAC2017ADR[111] for performing BEN. Li et al. [143] proposed the EhrBERT model, pre-trained on 1.5 million EHR notes, and evaluated it on three entity normalization corpora, namely the MADE corpus [105], NCBI disease corpus [60], and CDR corpus [62]. Authors observed that their models performed worse when the pre-training domain and fine-tuning task were distant.

6.2.6. Semantic text similarity (STS)

STS is an NLP task that measures the similarity between two pieces of text using a pre-defined metric. Xiong et al. [144] proposed a gated network to fuse one hot and distributed representations obtained from sentence-level features like inverse document frequency, sentence length, N-gram overlaps, and similarity metrics between two input sentences. Their fusion-gated BERT model was used on the clinical STS task of the BioCreative/OHNLP 2018 challenge [145]. Yang et al. [112] explored three models, BERT [20], XLnet [102], and RoBERTa [95], for clinical STS as a part of the 2019 n2c2/Open Health NLP challenge [113]. The models were pre-trained on a general STS dataset and fine-tuned on the clinical STS training partition. Among these, RoBERTa-large achieved the highest performance.

6.2.7. Automatic international statistical classification of diseases (ICD) coding

ICD codes are a set of alphanumeric designations to communicate diseases, symptoms, procedures, diagnoses, and abnormal findings in a universally accepted way among healthcare professionals. ICD coding involves recording the ICD codes associated with a patient’s visit. This coding process is often performed manually, which may result in documentation errors and consume a significant amount of time. Zhang et al. [146] proposed BERT-XML with multi-label attention to model 2292 ICD-10 codes from EHR notes [147]. Biswas et al. [148] used a transformer-based encoder architecture TransICD with a structured self-attention mechanism [149] to extract label-specific representations for multi-label ICD coding. Label distribution aware margin loss [150] was used to address the imbalance in ICD codes data. Transformer-based automatic ICD coding was used in clinical texts of Chinese [151], Spanish [152,153], Swedish [154], and Thai [155]. Silvestri et al. [156] used a Transformer Cross-lingual Language Model(XLM) [157] for automatic ICD coding by fine-tuning clinical texts in English and testing on clinical Italian text.

6.2.8. Neural machine translation (NMT)

Automatic NMT of biomedical data is essential to make essential healthcare information available to healthcare professionals overcoming language barriers. Tubay et al. [158] for the low-resourced biomedical NMT task used a Transformer model enhanced with a multi-source translation technique capable of exploiting multiple text inputs from the same language family. Berard et al. [159] proposed a multilingual neural machine translation (MNMT) model to translate biomedical text from 5 different languages (French, Spanish, German, Italian, and Korean) to English. The MNMT model is a variant of Transformer Big architecture with complex encoder capable of representing multiple languages. Liu et al. [160] proposed BioNMT Transformer model to translate domain specific biomedical vocabulary from foreign languages. The model is capable of semantic disambiguation of unknown words in the translation using external biomedical dictionaries. Wang et al. [161] used a Transformer large model with 20 encoder layers for biomedical translation shared task to translate German, French, and Spanish to English. Subramanian et al. [162] used a Transformer model for the same biomedical shared task at WMT to translate text from English to German and Russian. Their transformer model used a combination of model scaling, data augmentation with back-translation, knowledge distillation, model ensembling, and noisy channel re-ranking to perform the translation task.

7. Transformers for structured EHR data

Structured EHR data includes ICD codes for diagnoses, medication, vital signs, laboratory tests and other demographics collected every time a patient visits the hospital. These data are linked by an underlying temporal structure representing the cycle of diagnosis, medication/intervention, and potential patient readmissions. Furthermore, medication and diagnosis codes are derived from an ontological tree structure. Therefore, clinical tasks such as predicting future disease diagnoses, readmissions, or mortality rely on accurately representing the temporal and graphical structure of a patient’s EHRs. This challenge has led to three broad NLP tasks on structured EHR content that have been attempted in recent years using transformer networks.

7.1. Ontological structure learning

Previous studies have tried to learn the graphical structure inherent within the EHR using novel Transformer architectures. Choi et al. proposed the Graph Convolution Transformer (GCT) to jointly learn the relationships between diagnoses and medication codes while performing diagnosis-treatment classification [163]. They used conditional probabilities between medications and diagnoses calculated over the entire dataset to guide the attention maps in their Transformer network. Their model was validated on the eICU collaborative research dataset [164]. In contrast, Shang et al., 2019 explicitly used graph neural networks (GNN) for learning medical ontology embeddings and used these embeddings in a transformer to recommend future medications using the MIMIC-III dataset [165]. To leverage the entire dataset, they pre-trained G-BERT, a combination of GNN and BERT, on EHR data with only one admission. Peng et al., used a graph-based attention model (GRAM) to create ontological embeddings, which were then represented using multi-head self-attention to learn the ontological structure of medications within EHR [166].

7.2. Multi-modal data fusion

Previous studies have used Transformer networks to create joint embeddings amongst multiple data modalities, such as EHR and clinical notes. Darabi et al., used separate Transformer networks to create different representations for the clinical codes (ICD, drug, and procedure) and clinical notes and combined them into one “patient representation” [167]. They used this joint representation to predict future diagnoses, procedures, length of stay (LOS), readmission, and mortality. Studies have used joint-embeddings in BERT to predict rare diseases such as chronic cough [168] or depression [169]. Xu et al., proposed the use of multi-modal fusion architecture search (MUFASA), using an evolutionary algorithm to jointly search for the optimal architecture to represent subsets of EHR data and the optimal stage at which the individual embeddings will undergo fusion [170]. In contrast, Zhang et al., used a contrastive learning approach to increase the mutual agreement between different modalities for the same patient and increase the contrast for the same modality amongst different patients while jointly optimizing a prediction loss [171]. They showed that combining this representation with the BERT encoder predicted mortality and length of stay better than other baselines.

7.3. Predicting future diagnoses using ICD codes

BEHRT, an adaptation of BERT on EHR data, was trained from scratch using the masked language modeling task on sequential ICD codes and age to predict future diagnoses [172]. This model was developed primarily on the UK Clinical Practice and Research Datalink (CPRD) [173]. Recently, BEHRT was used to predict incident heart failure [174] and to perform causal inference [175]. The Hi-BEHRT model extended this by incorporating self-supervised pretraining by masking certain EHR data and certain time points in patients’ visitation history and creating localized feature aggregator Transformer embeddings fused at a later stage using global attention [176]. Hi-BEHRT performed better than BEHRT in predicting the onset of heart failure, diabetes, chronic kidney disease, and stroke. Compared to the BEHRT-based models, Med-BERT expanded the pretraining task to include prediction of prolonged length of stay and used a combination of ICD-9 and ICD-10 codes to create their model, which was subsequently evaluated on predicting diabetes and heart failure [177]. Another model, HiTANet [178], explicitly included a time vector to represent the time elapsed between consecutive visits. The time embedding was combined with the original visit embedding and used as key values in a global attention block to represent the most significant time points in a patient’s medical history. They tested their model efficacy in predicting future diagnoses of three disease-specific datasets. The RAPT model combined an explicit time-span information vector with additional pre-training tasks such as similarity prediction and reasonability check to address data insufficiency, incompleteness, and short sequence problems inherent in EHR data [179]. They evaluated their model for predicting pregnancy outcomes, risk period, and the diagnoses of diabetes and hypertension during pregnancy.

8. Transformers in medical imaging

8.1. Medical image segmentation

Image segmentation is a dense pixel classification task that captures the complex interactions between individual pixels of an image. Unlike general-purpose image segmentation, medical image segmentation suffers from a lack of large datasets, requires the context of surrounding anatomical structures, and must account for inter-patient anatomical variabilities. Several data modalities, such as X-ray, Ultrasound, Magnetic Resonance Imaging (MRI), Computed Tomography (CT), Positron Emission Tomography (PET), and microscopy, can benefit from medical image segmentation. Before Transformers, the U-net architecture, proposed by Ronneberger et al. [180], was the prominent architecture for medical image segmentation. The U-net model is a Convolutional Neural Network (CNN). Convolutional layers are limited in long-range feature modeling. This is because the receptive field of convolutional filters increases linearly; therefore, only the deepest convolutional layers have the global context of an image. Although incorporating dilation and stride into convolution can address the limitations of long-range dependencies to some extent, it results in an unavoidable tradeoff between global and local information. On the contrary, the self-attention mechanism in Transformer layers can model the global context of images, irrespective of layer depth. A comprehensive list of transformer-based models for segmentation is provided in Table 3.

Table 3.

Transformers for medical image segmentation,

Reference	Title	Datasets	Task	Modalities
Chen et al. [181]	TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation	Synapse [182], ACDC [183]	Multi-organ segmentation, Cardiac segmentation	CT, MRI
Valanarasu et al. [184]	Medical Transformer: Gated Axial-Attention for Medical Image Segmentation	Brain Segmentation, GLAS [185], MoNuSeg [186,187]	Brain-anatomy segmentation, Gland segmentation, Nucleus segmentation	Ultrasound, Microscopy
Chang et al. [188]	TRANSCLAW U-NET: CLAW U-NET WITH TRANSFORMERS FOR MEDICAL IMAGE SEGMENTATION	Synapse [182]	Multi-organ segmentation	CT
Hatamizadeh et al. [189]	UNETR: Transformers for 3D Medical Image Segmentation	BCV [182], MSD [190]	Multi-organ segmentation	CT
Gao et al. [191]	UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation	M&Ms [192]	Cardiac segmentation	MRI
Zhang et al. [193]	TransFuse: Fusing Transformers and CNNs for Medical Image Segmentation	Kvasir [194], CVC-Clinic [195], CVC-Colon [196], EndoScene [197], ETIS [198],	Polyp segmentation, Skin lesion segmentation, Hip segmentation. Prostate segmentation	Colonoscopy,
Xie et al. [199]	CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation	BCV [182]	Multi-organ segmentation	CT
Cao et al. [200]	Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation	Synapse [182], ACDC [183]	Multi-organ segmentation, Cardiac segmentation	CT MRI
Huang et al. [201]	MISSFormer: An Effective Medical Image Segmentation Transformer	Synapse [182], ACDC [183]	Multi-organ segmentation, Cardiac segmentation	CT MRI
Zhang et al. [202]	Pyramid Medical Transformer for Medical Image Segmentation	GLAS [185], MoNuSeg [187], HECKTOR [203]	Gland segmentation, Nucleus segmentation Tumor segmentation	Microscopic images, CT/PET
Ji et al. [204]	Multi-Compound Transformer for Accurate Biomedical Image Segmentation	Pannuke[205], CVC-Clinic [195], CVC-Colon [196], ETIS [198], Kvasir [194], ISIC2018 [206]	Cell segmentation, Polyp segmentation, Skin lesion segmentation	Pathology, Colonoscopy, Dermoscopy
Lin et al. [207]	DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation	CVC-Clinic [195], CVC-Colon [196], EndoScene [197], ETIS [198], GLAS [185], Kvasir [194], ISIC2018 [206]	Polyp segmentation, Skin lesion segmentation, Gland segmentation, Nucleus segmentation	Pathology, Colonoscopy, Dermoscopy
Li et al. [208]	Medical Image Segmentation Using Squeeze-and-Expansion Transformers	REFUGE2020 [209], Drishti-GS [210], RIM-ONE v3 [211], Kvasir [194]	Optic disc and cup segmentation, Polyp segmentation, Brain tumor segmentation	Colonoscopy, MRI, Fundus images
Yun et al. [212]	SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation	Choledoch [213]	Pathology segmentation	Pathology
Xu et al. [214]	LeViT-UNet: Make Faster Encoders with Transformer for Medical Image Segmentation	Synapse [182], ACDC [183]	Multi-organ segmentation, Cardiac segmentation	CT MRI
Wang et al. [215]	Transbts: Multimodal brain tumor segmentation using transformer	BraTS 2019 [216,217], BraTS 2020 [216,217]	Brain tumor segmentation	MRI
Chen et al. [218]	TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation	ISIC 2018 [206], JSRT[219], Montogomery [220], NIH [221], Clean-CC-CCII [222], GLAS [185], Bowl [223]	Chest X-ray segmentation, Skin lesion segmentation, Nucleus segmentation, Gland segmentation	X-ray, Histology, CT
Petit et al. [224]	U-net transformer: self and cross attention for medical image segmentation	TCIA, Internal dataset	Abdominal organ segmentation	CT
Yan et al. [225]	AFTer-UNet: Axial Fusion Transformer UNet for Medical Image Segmentation	BCV [182], Thorax-85 [226], Segthor [227]	Multi-organ segmentation, Thoracic segmentation	CT
Guo et al. [228]	A Transformer-Based Network for Anisotropic 3D Medical Image Segmentation	MSD [190]	Lung cancer segmentation	CT
Sun et al. [229]	HybridCTrm: Bridging CNN and Transformer for Multimodal Brain Image Segmentation	MRBrainS [230], iSEG-2017 [231]	Brain tissue segmentation,	MRI
Tang et al. [232]	Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis	BTCV [182], MSD [233]	Multi-organ abdominal segmentation	CT
Zhang et al. [234]	TiM-Net: Transformer in M-Net for Retinal Vessel Segmentation	STARE [235], CHASEDBI [236], DRIVE [237]	Retinal vessel segmentation	Color images
Wang et al. [238]	Auxiliary Segmentation Method of Osteosarcoma in MRI Images Based on Denoising and Local Enhancement	In house dataset	Osteosarcoma segmentation	MRI
Shen et al. [239]	Dilated transformer: residual axial attention for breast ultrasound image segmentation	BUSIS [240]	Breast segmentation	Ultrasound
Thanh Duc et al. [241]	ColonFormer: An Efficient Transformer Based Method for Colon Polyp Segmentation	Kvasir [194], CVC-Clinic[195], CVC-Colon [196], CVC-T [197], ETIS [198]	Polyp segmentation	Colonoscopy

CT: Computed Tomography; MRI: Magnetic Resonance Imaging; PET: Positron Emission Tomography

8.1.1. CNN-transformer hybrids

TransUNet, proposed by Chen et al. [181], is shown in Fig. 8 and was one of the earliest examples. TransUNet uses a CNN to downsample the input image before providing it to a Transformer encoder, creating a global contextualized deep representation of the image. This representation is subsequently passed through a cascaded up-sampler to convert it into the full-resolution segmented output image. The idea of using a Transformer as a U-net encoder to learn long-range dependencies was subsequently adapted by multiple studies such as TransClaw U-Net [188], BiTr-UNet [242], Bi-FPN-UNet [243], and Weaving Attention U-Net [244]. UNet-Transformer used MHCA in skip-connections between the encoder and the decoder to recover finer spatial features [224]. LeViT-Unet [214] integrated LeVIT [245] into the downsampling block of U-net. TransAttUnet [218] used a novel self-aware attention module with both Transformer self-attention and global spatial attention.

Fig. 8.

Overview of TransUNet architecture. a) Schematic of Transformer encoder b) TransUNet architecture. Source: The figure was adapted from [181] without modifications.

For 3D medical image segmentation, UNETR [246] used ViT-B16 [247] as the encoder instead of CNN while retaining the U-shaped network design. TransBTS used 3D CNN blocks as an encoder to model spatial information, a Transformer encoder to capture long-distance dependencies, and a decoder to model volumetric data in MRI scans [215]. CoTr concatenated CNN feature maps at different scales using positional encoding and passed them into stacked Deformable Transformer encoder blocks [248]. Deformable Transformer computed attention over a local region around reference points instead of global self-attention, reducing computational complexity. The authors showed that this methodology outperformed other CNN-Transformer hybrid models on the BCV dataset [182] that covers 11 major human organs. SpecTr [212] used adaptively sparse Transformer blocks [249] to remove redundant/noisy bands of spectral information in the Transformer encoder while segmenting hyperspectral images. This study also used 3D CNN encoders in combination with Transformer encoders in a U-Net fashion. The nnFormer [250] is a 3D Transformer for volumetric image segmentation that uses interleaved convolutional and local/global self-attention operations coupled with skip attention between the encoder and decoder to achieve better performance over other CNN-transformer hybrid models in three datasets [182,183,233]. Tang et al. [232] developed a new 3D Transformer-based model named Swin UNEt Transformer (Swin UNETR) with a hierarchal encoder for self-supervised pre-training using five public CT datasets. The model contains a Swin Transformer encoder that directly utilizes 3D patches and is connected to a CNN-based decoder via skip connections at different resolutions. The model was fine-tuned and validated using the BCV dataset [182] and the Medical Segmentation Decathlon (MSD) dataset [233]. These studies reflect effective ways of combining convolutions with attention in medical image segmentation.

8.1.2. Transformer-Only U-Nets

UTNet [191] introduced Transformer self-attention into the encoder and decoder to capture long-range dependencies at different scales. Swin-Unet [200] used pure Swin Transformer [251] blocks. The DS-TransUNet model used a dual-branch Swin Transformer in the encoder to extract feature representations at multiple scales, and Transformer Interactive Fusion (TIF) blocks to establish global interactions between them [207]. Valanarasu et al. [184] proposed Medical Transformer (MedT) with a gated axial attention layer along with local and global branches (LoGo), adapted based on position-sensitive axial attention [252] to influence positional bias on small-scale medical datasets. Karimi et al. developed a convolution-free 3D segmentation framework using pre-trained vanilla Transformer encoder, which performed better than CNN models on three proprietary datasets [253].

8.1.3. Non U-Net transformer models

Zhang et al. [234] developed the TiM-Net model based on M-Net [254] with diverse attention mechanisms and weighted side output layers for retinal vessel segmentation. The model was validated on three public retinal image datasets: STARE [235], CHASEDBI [236], and DRIVE [237]. Wang et al. [238] proposed an auxiliary segmentation method for osteosarcoma detection in MRI images based on denoising and local enhancement. For noise removal, the authors used the Eformer [255]. Duc et al. [241] developed a network called ColonFormer for polyp segmentation from endoscopic images on Kvasir [194], CVC-Clinic DB [195], CVC-Colon DB [196], CVC-T [197], and ETIS-Larib Polyp DB [198] datasets. The model uses Mix Transformer [256] as the encoder backbone, a hierarchical Transformer encoder that can represent both high and low-resolution features. It also includes efficient Self-Attention to reduce the computational complexity of self-attention layers.

8.2. Medical image registration

Image registration is the process of transforming data from multiple datasets into one coordinate system. Registration is essential for comparing, analyzing, or integrating data obtained from various sources, viewpoints, times, or sensors [257]. An example of registration is aligning CT and MRI scans of patient captured obtained from different view points and varying patient head orientation. In image registration task source and target images are provided as input to deep learning model to estimate the spatial transformation parameters between the images. Recent approaches have incorporated attention-based Transformer models for this task.

Chen et al. proposed one of the earliest Transformer based architectures, VIT-V-Net [258], to combine the vision Transformer (ViT) [247] and V-Net [259], a CNN architecture. The ViT is used to extract the features from the fixed and moving images, followed V-Net style decoder to predict the displacement field. Wang et al. [260] developed TUNet to incorporate ViT [261] into the U-Net [180] architecture to extract global and local features from moving and fixed images. Mok et al. [262] developed a fast, robust learning-based algorithm called C2FViT for 3D affine medical image registration. C2FViT leverages global connectivity, the convolutional vision Transformer locality, and a multi-resolution strategy. Both papers evaluated their models on brain template-matching normalization and atlas-based registration using the OASIS [263] and LPBA [264] datasets. Tulder et al. proposed pixel and token-wise cross-view attention to integrate multiple views in mammography and X-ray imaging [265] using CBIS-DDSM [266] and CheXpert [267] datasets.

Chen et al. proposed TransMorph [268], a modified U-net architecture that incorporates Swin Transformer [251] blocks in its downsampling branch for unsupervised affine and deformable image registration on the IXI [269] dataset. Transformer blocks enabled the estimation of deformation uncertainty while preserving the registration performance. Zhu et al. [270] proposed the Swin-VoxelMorph. This unsupervised learning model applies a hierarchical Swin Transformer [251] as the encoder to extract contextual information and a symmetric Swin Transformer-based decoder with a patch-expanding layer to perform up-sampling to estimate the registration fields. The authors validated the model on ADNI [271] and PPMI [272] datasets.

8.3. Medical image captioning and report generation

Expert medical professionals typically interpret biomedical images, documenting their findings as medical reports, a time-consuming task. Automated medical report generation can reduce the workload and reduce human error. The image captioning/report generation tasks involve generating a textual description of a provided visual input. The input image is processed through a deep learning model to extract relevant feature information, which is fed into a language model to a coherent and contextually appropriate textual representation in the form of a sequence of words.

Hou et al. [273] proposed the RATCHET model, a medical Transformer, to generate medical text reports from chest X-rays. The authors used the MIMIC CXR v2.0.0 dataset [274] with over 300,000 chest radiograph images and free-text radiology reports. Free text reports were tokenized using the byte pair encoding approach [275]. The RATCHET architecture follows the encoder-decoder architecture, but the encoder is a CNN model, DenseNet-121 [276], whereas the decoder is the vanilla Transformer decoder. The output features of the DenseNet-121 encoder are provided as input to the second attention block of the Transformer decoder, whereby the network learns context from the radiography image against the input text report. Free text tokens are shifted right and provided as input to the decoder to predict the next token. Nicholson et al. used a pretrained ViT encoder and a pretrained PubMedBERT decoder to solve the 2021 ImageCLEFmed Caption task [277]. Their model was fine-tuned on the ROCO dataset [278] and tested on PadChest [279], CheXpert [267], ChestX-ray14 [280], and MURA [281] datasets. Alfarghaly et al. [282] used conditioned self-attention, where new key and value parameters were introduced to project the encoder’s output to the decoder’s attention space. The authors extracted visual and semantic features using Chexnet [283], a Densenet121 model, and pre-trained word2vec embeddings, respectively. For the training and validation of the model, they used the IU-Xray dataset [284]. You et al. [285] developed an AlignTransformer for chest X-ray images consisting of two modules: Align Hierarchical Attention (AHA) and Multi-Grained Transformer (MGT). The AHA module was used to align visual regions and disease tags. Features from the AHA module were provided as input to the MGT module. The MGT module adaptively exploited multi-grained disease-grounded visual features to determine the importance of visual features for each target word. The authors used two publicly available datasets: IU-Xray [284] and MIMIC-CXR [286]. Pahwa et al. [287] developed a memory-driven Transformer model called MedSkip for report generation. MedSkip consists of the standard Transformer encoder and a relational memory decoder. It was trained on Pathology Education Informational Resource (PEIR) Gross dataset [288] and IU X-Ray [284] datasets. Li et al. developed a Cross-modal clinical Graph Transformer (CGT) to incorporate expert knowledge into ophthalmic report generation [289]. The model first restores a sub-graph from the clinical graph and injects clinical relation triples into the visual features as prior knowledge. Reports are predicted using the encoded cross-modal features using a Transformer decoder. The CGT model was trained and validated on an ophthalmic report generation dataset called FFA-IR [290].

8.4. Visual question answering (VQA)

VQA is a computer vision task where a question is posed, and the answer must be inferred from an image. In the medical domain, VQA can be used to extract information from medical images to assist in making a diagnosis. Ren & Zhou, 2020 [291] developed the CGMVQA model, which modified the original Transformer using layer normalization before the MHSA and FCFN layers. The model was trained and validated on the ImageCLEF 2019 VQA-Med data set [292]. The CGMVQA can interchangeably deploy a classification or a generative mode by changing the output layer and loss function while retaining the same architecture. While in the classification mode, the model can predict a yes-no modality, in the generative mode, the model uses masked answers to predict the next word in a sentence. Naseem et al. [293] introduced the TraP-VQA model to answer medical questions presented in pathology images. This model embedded low-level visual features extracted using a CNN, low-level language features extracted using a domain-specific language model, and the Transformer layer to learn the contextualized representation between the two to solve the VQA task. The authors used the public PathVQA dataset [294] to train and validate their model. Sharma et al., 2021 v developed an attention-based multimodal model called MedFuseNet, using BERT for question feature extraction, which was found to be more effective than XLNet [102] and two datasets for training: ImageCLEF 2019 MED-VQA [292] and PathVQA datasets [294].

8.5. Image synthesis

Medical image synthesis aims to replace or bypass an imaging procedure constrained by time, cost, and labor or to prevent exposure to harmful ionizing radiation from some imaging modalities. It involves synthesizing medical images of a target modality from source images such as synthesizing MRI scan from CT or vice-versa. Dalmaz et al. [295] proposed a novel encoder-decoder-based generative adversarial network (GAN) model RESVIT for synthesizing missing sequences in multi-contrast MRI and pelvic CT images from source MRI images. The network architecture consists of a CNN encoder, decoder, and aggregated residual Transformer to learn global representations. RESVIT model synergistically fuses local and global feature representations to achieve superior image synthesis quality. Other GAN-based [296] models, such as CycleGAN [297] and CyTran [298], have been used to create contrast CT scans from non-contrast CT scans and vice versa. The CyTran architecture incorporates convolutional upsampling, convolution downsampling, and a convolution Transformer block to perform the translation. Kamran et al. [299] proposed VTGAN to combine two generators for examining local and global features with ViT [247] in a semi-supervised manner to synthesize Fluorescein Angiography images [300] while predicting retinal degeneration. VTGAN successfully synthesized angiograms from fundus images and proved robust on spatial and radial transformations.

Yan et al. created MMTrans [301] using a Swin-Transformer [251] as both a generator and registration network and a CNN as the discriminator. The generator was used to generate images with the same content as the source modality and the same style as the target modality. In contrast, the discriminator was used to measure the similarity between the original target modality images and those synthesized by MMTrans. Hu et al. proposed a double-scale graph neural network (GNN) [302] combined with a Transformer to learn long-range dependencies from global features, while for local features, they used CNN. It outperformed established baselines in the IXI dataset. Liu et al. introduced a multi-contrast multi-scale Transformer (MMT) [303], by using missing data imputation as input and proposed a Multi-contrast Shifted Window (M-Swin) to capture intra- and inter-contrast dependencies.

PTNet [304], proposed by Zhang et al., synthesizes infant MRI [305] scans using a U-net [180] based architecture that incorporates a performer [306] encoder and a decoder with linear space and time complexity. PTNet outperformed previous CNN-based approaches and had an execution time of 30 slices per second. Zhang et al. further extended PTNet to 3D MRI as PTNet3D [307] and evaluated it on high-resolution Developing Human Connectome Project (dHCP) [305] and longitudinal Baby Connectome Project (BCP) datasets [308].

8.6. Image reconstruction

Image reconstruction aims to reconstruct high-quality medical images with minimal cost and risk to the patient.

8.6.1. Computed tomography (CT)

Low-dose computed tomography (LDCT) imaging for clinical diagnosis uses a reduced dose of X-ray radiation compared to conventional CT scans. However, LDCT is prone to noise, which affects the scan quality. Zhang et al. proposed TransCT [309] to enhance the quality of LCDT images using the AAPM-Mayo LDCT dataset [310]. The input image was decomposed into low-frequency and high-frequency components, and then the content, texture, and high-frequency embeddings were fed to the TransCT model to obtain refined high-frequency textural features. Luthra et al. proposed Eformer [255] by combining learnable edge-enhancement convolutions called Sobel filters and the LeWin transformer [311] in denoising LDCT images for detecting metastatic liver lesions (AAPM-Mayo dataset) [310]. Wang et al. [312,313] proposed convolution-free transformer-based encoder-decoder dilation networks (TED-net) using vanilla transformer blocks for LDCT denoising. Instead of an image, a few approaches used informative sinograms generated by restoration modules from origin LDCT images for reconstruction using Transformer-based models [314–317].

8.6.2. Magnetic resonance imaging (MRI)

Korkmaz et al. proposed an MRI reconstruction model based on a zero-shot learned adversarial vision Transformer named SLATER [318] to overcome the data size limitation. Inspired by Deep Image Prior (DIP) [319], they replaced the CNN backbone of DIP with a cross-attention Transformer and outperformed DIP on the IXI dataset [269] and fastMRI dataset [320]. Feng et al. [321,322] introduced a multi-task framework T2Net, to share the representations between reconstruction and super-resolution branches. Furthermore, they extended to multi-modalities (MTrans), aiming to learn more knowledge from MRI using both branches. Fang et al. proposed a cross-modality high-frequency Transformer (Cohf-T) [323] for super-resolving, low-resolution MR images. Guo et al. proposed a lightweight recurrent Transformer model ReconFormer [324], which includes pyramid transformer layers [325] to capture intrinsic multiscale information and feature correlation through the recurrent states. Li et al. proposed McMRSR [326], a Transformer based network to model long-range dependencies between reference and target images and aggregate multiscale matched features to reconstruct a target MR image. Few approaches use raw K-space signals of MRI scans instead of final MRI images as they contain learnable information for MRI reconstruction [320,327–330]. Hu et al. introduced a Transformer-enhanced Residual-error AlterNative Suppression Network [331], which included a regularization term to improve the contribution of high-frequency information during inference. Fabian et al. [332] proposed HUMUS-Net, a two-level hybrid CNN Transformer architecture for MRI reconstruction using the fastMRI dataset [320]. Huang et al. [333] proposed a GAN [296] based on Swin-Transformer [251] named ST-GAN, which preserved edge and texture features. Swin-Transformer inspired shifted window attention became the go to Transformer architecture for many studies targeting MRI reconstruction [328,334–336].

8.6.3. Positron emission tomography (PET)

PET is an imaging technique that measures emissions from radioactively labeled chemicals injected into the bloodstream. PET scans can measure metabolic activity and other biochemical functions. Unfortunately, PET suffers from a poor signal-to-noise ratio, and its reconstruction requires denoising low-quality PET images to create high-quality ones. Luo et al. proposed a GAN based Transformer model, Transformer-GAN [337], for PET reconstruction with CNN(Encoder)-Transformer-CNN(Decoder) architecture to take advantage of spatial information and long-range dependencies from CNN and transformers, respectively. Fu et al. extended their transGAN-SDAM [338] for fast 2.5D-based L-PET. The transGAN generates higher quality F-PET images, followed by the SDAM module, which combines spatial information of an F-PET slice sequence to generate whole-brain F-PET images. Jang et al. proposed Spach Transformer [339] that can leverage spatial and channel-wise information based on local and global MHSA, which outperformed baselines on different PET tracer datasets of 18F-FDG, 18F-ACBC, 18F-DCFPyL, and 68GaDOTATATE.

9. Transformers for critical care

9.1. Predicting long-term adverse outcomes

Transformers have been used to predict adverse outcomes after critical care such as recurrence or death. Yang et al., 2021 predicted a 60-day and 90-day response to targeted immunotherapy of patients with non-small cell lung cancer (NSCLC) using asynchronous clinical time series consisting of chest CT scans, and blood tests, and patient characteristics using an attention module called Simple Temporal Attention [340]. The model predicted which patients would have long-term durable survival gains under an immunotherapy regimen. Similarly, in 2021 for colorectal cancer, Ho et al. used Transformer encoders to extract features from sequential carcinoembryogenic antigen (CEA) measurements. It combined CEA measurement features with deep representations of tabular features such as tumor sites, number, dates, and dosage of chemotherapy to predict recurrence [341]. They modified the Transformer to incorporate 1D convolutions prior to localized self-attention [342]. Their model outperformed commercial diagnostic tests of colorectal cancer recurrence. Non-clinical population-level claims data has also been modeled using multi-headed self-attention to predict relapse after surgery [343,344]. These studies utilized the French national health insurance database (SNIIRAM), consisting of health-insurance claims entries of 65 million individuals [345].

9.2. Surgical instruction generation

Intra-operative surgical assistance AI systems need to solve the task of automatic surgical instruction generation. Zhang et al., 2021 used a Transformer-backboned encoder-decoder network combined with self-critical reinforcement learning (RL) to jointly model surgical activity and relationships between visual information and textual description [346]. They used the Database for AI Surgical Instruction dataset (DAISI) to evaluate their model [347]. The authors used a combination of machine translation and image-captioning criteria to evaluate their models, such as BLEU [348], Rouge-L [349], METEOR [350], and CIDEr [351], and SPICE [352]. The combination of Transformer with RL beat baselines comprising LSTM-based fully connected and soft-attention models.

10. Transformers for social media data in public health

In recent years, using social media data has gained prominence in different areas of public health [353–356]. Transformers have been applied to social media data for addressing several public health problems, such as monitoring adverse drug reactions [357,358], monitoring mental health [359], categorizing vaccine confidence [360], and locating disease hotspots [361]. In this section, we present the models and their performance on social media datasets.

10.1. Monitoring adverse drug events (ADEs)

ADEs, refers to an undesired, unpleasant, or dangerous reaction to a medication [362], which has been found to be underreported; thus, researchers have recently used social media to improve ADE monitoring [363]. The main steps in monitoring ADRs using social media posts are text classification to find ADE mentions, followed by extracting the ADE concept and mention from the classified text. Breden et al. [357], preprocessed the Twitter dataset from Social Media Mining for Health (SMM4H) 2019 Competition [364] using the lexical normalization [365] method. The best-performing model was an ensemble of fine-tuned BERT, BioBERT [54] and ClinicalBERT [59]. Sakhovskiy et al. [366] used a more recent dataset provided by SMM4H 2021 [367] for task1, classifying English tweets by concatenating the RoBERTa [95] and ChemBERTa [368] models. For task2, Russian tweets classification performed by concatenating the EnRuDR-BERT [369], and ChemRoBERTA [362] cross-attention. Hussain et al. [370] proposed an end-to-end system based on transfer learning using one prediction head for the text classification and another for labeling the adverse drug responses. The authors fine-tuned BERT with a modular Framework for Adapting Representation Models (FARM) and present the FARM-BERT framework, which outperforms competing models on TwiMed-Twitter [371], Twitter [372], PubMed [373], and TwiMed-PubMed [371] datasets. The framework FARM-BERT supports multitask learning by combining multiple prediction heads, making training of the end-to-end systems easier and computationally faster. Raval et al.[358], tackled the same ADE classification problem; however, they framed it as a sequence-to-sequence problem and used the pre-trained T5 model architecture [374] on multiple datasets (SMM4H [375], CADEC [376], ADE corpus v2 [373], WEB-RADT [377], SMM4H-French [375]). The authors further expanded the proportional mixing and temperature scaling training strategies described in [378] to handle multi-dataset and present relative improvement on the F-1 score.

10.2. Monitoring depression

A large-scale depression dataset on Twitter was presented by Zhang et al. [359], used Transformer-based models to identify users suffering from depression using their everyday speech. The importance of psychological text features was also studied when performing depression classification. Results on the fluctuating depression levels for different groups were also presented. Matero et al. [379] used pretrained BERT embeddings to encode this information. Kabir et al. [380] presented a dataset observing the severity of depression in tweets and reported baseline results using BERT and DistilBERT [381].

10.3. Monitoring diabetes

Large-scale Twitter data concerning diabetes-related tweets have been collected and used to identify cause-effect relationships [382]. They used a pre-trained BERTweet model [383] to detect causal sentences and a combined BERT+ Random Field Generator model to extract potential cause-effect relationships.

10.4. Categorizing vaccine confidence

Social media plays a crucial role in gauging public discourse on topics such as vaccine effectiveness [384]. It provides a proxy to analyze vaccination apprehensions and study the barriers to successful vaccinations [385]. Kummervold et al. [360] used domain-specific BERT model to assess the social media stance towards vaccination during pregnancy on a dataset of 2722 unique tweets. The model was able to achieve accuracy of a trained human annotator in categorizing the stance, outperform other models and human coders in some cases.

10.5. Locating disease hotspot

It is essential to detect disease outbreaks while simultaneously reducing reporting lag time. This can provide another source of data to complement traditional surveillance approaches. Alsudias et al. [361] performed a multi-label classification task to identify tweets of infected individuals in the Arabic-speaking world. The authors propose a combination of binary relevance, classifier chains, label power set, multi-label adapted k-nearest neighbors (MLKNN) [386], support vector machine with naive Bayes features (NBSVM) [387], BERT and AraBERT (transformer-based model for Arabic language understanding) [388]. The proposed model achieved an F1 score of up to 88% in the influenza case study and 94% in the COVID-19. It is shown that including informal terms and non-standard terminology (e.g., the slang term of influenza, symptom, prevention, treatment, infected with) in the encodings improved the performance by as much as 15%, with an average improvement of 8%. The proposed geolocation detection algorithm performed moderately in predicting the location of users according to their tweet content.

11. Monitoring bio-physical signals

Transformers have been used to model physical activity, Electroencephalogram (EEG), and Electrocardiogram (ECG) signals. In the following sections, we review these works.

11.1. Human activity recognition (HAR)

Human Activity Recognition (HAR) is a proliferating field of research owing to the recent rise of wearables, smartphones, and Internet of Things (IoT) devices. Some studies have used multimodal self-attention to fuse features from various modalities [389,390]. They studied sequences of human movements through multimodal data (such as RGB, depth, and skeletal data) [391–393] or modeled human activity through accelerometers and gyroscopes [394–397]. Spatiotemporal bone and joint sequences from skeleton data have been modeled using multi-scale Transformers on multiple datasets [398–401]. Owing to the lack of simple augmentation strategies of longitudinal sensor data, Ramachandra et al. used Transformer-GAN to provide a speedup over existing Recurrent-GAN [402].

11.2. Electroencephalogram (EEG)

Electroencephalogram (EEG) is a widely used noninvasive measurement of brain activity. Transformers have been used to classify visual or motor imagery using EEG signals [403]. It has been shown that extensive self-supervised pre-training using contrastive loss can help Transformer models represent EEG data collected using different hardware while performing different tasks [404]. Pretraining was conducted using the Temple University Hospital EEG Corpus [405] and downstream analyses were done using a battery of smaller datasets [406–408]. Cross-modal Transformers have been used to find contextualized embeddings representing associations between auditory attention detection and EEG signals [409]. This can disentangle sources of brain activity at different time points while the subject is attending to multiple sound sources simultaneously. This study was conducted on the Denmark Technical University (DTU) dataset [410,411] Finally, a 2D Transformer was used to capture local self-similarity, and feed-forward connections were used to capture global self-similarity to create a novel denoising system for 1D EEG signal [412] using another publicly available dataset [413].

11.3. Electrocardiogram (ECG)

Electrocardiogram (ECG) signals alone and combined with other sensory information were used to predict stress in subjects using Transformers [414,415]. The Wearable Stress and Affect Detection (WESAD) and SWELL Knowledge Work (SWELL-KW) are publicly available datasets used for this purpose [416,417]. A transformer network embedded inside a CNN architecture has been used to classify arrhythmia [418].

12. Transformers for biomolecular sequences

Biomolecular sequences can represent genomic, proteomic, and drug data. As sequence translation models, transformers have been widely used to model the relationships between anomalous biological sequences and related diseases. Moreover, drug/protein synthesis or gene sequence alignment problems have been treated through the lens of machine translation, where the Transformer is the model of choice.

12.1. DNA

Gene Transformer, which consists of a multi-head self-attention module, detects lung cancer subtype biomarkers [419]. It consists of two 1D convolutional layers before the MHSA layer to extract low and moderate-level features. A previous study utilized RNA-sequencing values from lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC) datasets from the Cancer Genome Atlas project [420]. Clauwaert et al. 2020 introduced an attention method optimized for nucleotides on top of the Transformer-XL architecture [421]. This attention module included a 1D convolutional layer that extracted overlapping DNA segments of length k called k-mers from the original DNA sequences’ query, key, and value matrices. The authors solved three problems, including a) annotating the transcription start site (TSS), b) annotating the translation initiation site (TIS), and c) recognizing 4mC methylation sites using the following datasets – RegulonDB [422], Ensembl [423], and MethSMRT [424], respectively. A following study utilized comparative TSS annotations from multiple datasets, including RegulonDB [422], Etwiller, et al., 2016 (Cappable-seq) [425], Yan et al., 2018 (SMRT-Cappable-seq) [426], and Ju et al., 2019 (Send-seq) [427]. In another study, the Transformer-XL network was highly biased toward attending to promoter regions and transcription factor binding sites near the gene under question [428]. Another network, DNABERT was used to predict transcription factor binding (TFB) sites, including proximal and core promoter regions, splice sites, and genetic variants [429]. Reference human genome GRCh38.p13 primary assembly from GENCODE Release 33 [430] was used for pre-training, TATA, and non-TATA promoter data from Eukaryotic Promoter Database (EPDnew) [431] for promoter prediction and ENCODE 690 ChIP-seq profiles from UCSC genome browser [432] were used for predicting TFB sites. Enhancers are regulatory elements that activate promoter transcription over large distances independently of orientation [433]. BERT, pre-trained with masked language modeling (MLM) and the next sentence prediction tasks, was combined with 2D convolutions to predict transcription enhancers using a dataset describing an enhancer sequencer from nine cell lines [434,435].

12.2. Protein

Transformers can either predict global properties of protein such as type, function, or cellular localization or infer local properties of selected protein residues such as 2D/3D structure or post-translation modifications (such as phosphorylation and cleavage sites) [436]. The recent success of AlphaFold in protein structure prediction problems [437] has significantly changed the domain [438], although recent advances have primarily included fine-tuning pre-trained deep models for learning with small datasets [436].

12.3. Molecular drugs

Transformer have been utilized for the prediction of molecular drugs as follows.

12.3.1. Drug-drug synergy

One of the most useful applications of Transformer networks is in the finding of synergistic combinations of drugs for the treatment of diseases which cannot be cured by a single molecule. The classic example of this is cancer. In cancer, drug combinations alleviate drug resistance and improve therapeutic efficacy. However, the rapidly growing number of anti-cancer drugs makes it extremely resource intensive to search the entire space of synergistic drug combinations. This is where computational models like the Transformer are useful. The TranSynergy model constructed a Transformer model of the cellular effect of drug combinations on different gene-cell line combinations by modeling cell-line gene dependency, gene-gene interaction, and genome-wide drug-target interaction, thereby introducing mechanistic knowledge into the model [439]. The study utilized a large drug synergy score dataset [440] and drug target profiles from DrugBank[441] and ChEMBL[442]. TranSynergy outperformed the SOTA and predicted multiple novel synergistic drug combinations for treating ovarian cancer. Kim et.al., 2020 used multi-task transfer learning to study drug synergy in understudied tissues to overcome data scarcity problems [443]. The authors used a multi-head Transformer network to create an embedding of the Simplified Molecular-Input Line-Entry System (SMILES) representation of drugs. TP-DDI presents a completely end-to-end Transformer pipeline with pretrained BioBERT weights for drug recognition and drug-drug interaction (DDI) classification [444]. This study is conducted on the DDI Extraction 2013 corpus [86] which consists of a list of semantically annotated documents with sentences referring to drugs and DDIs from the DrugBank database and MedLine abstracts.

12.3.2. Drug synthesis

Transformers have been used to convert the task of target-driven de novo drug-synthesis into a neural machine translation task that converts an amino acid sequence into the chemical formula of its binding drug [445]. This method needs neither any prior information about the drug structure nor the 3D structural information of the protein target. The study used a dataset of binding affinity between proteins and drug-like molecules from the BindingDB database [446]. Synthesized drugs were evaluated on active properties like the number of hydrogen donors/acceptors, molecular weight, length, total polar surface area, number of rotatable bonds, and drug-likeness. Born et.al., 2021 studied the synthesis feasibility of drugs for use against the SARS-Cov-2 virus using a transformer-based retrosynthesis prediction engine [447] consisting of two molecular transformers [448]. They operate on a SMILES representation of a molecule to predict best routes for its synthesis [449]. This information was further utilized by another Transformer model to predict the optimal synthesis protocol using a text representation of the synthesis steps [450]. The approach incorporated variational autoencoders and reinforcement learning to automatically learn molecules that target ACE2, a surface receptor on human epithelial cells that allows entry of the SARS-Cov-2 virus [449].

12.3.3. Drug-target interactions

In-silico drug discovery is driven by computational models of drug-target interactions. Huang et al. developed the Molecular Interaction Transformer, which models the interaction space between the most common substructures of molecules and drugs [451]. These substructures were discerned using Frequent Consecutive Sub-sequence algorithm on protein sequences from UniProt dataset [452] and drug SMILES strings from ChEMBL [453]. In this work, a Transformer encoder is used to create contextualized embeddings of protein and drug substructures separately which are multiplied to capture their interaction strengths. A CNN extracts higher order interactions from joint space. Three datasets were employed to learn the transformer and CNN weights-MINER DTI from BIOSNAP [454], BindingDB [455] and DAVIS [456].

Manica et al., 2021 proposed an anticancer drug sensitivity model using drug SMILES sequences, gene expression profile of tumors, and protein-protein interaction networks.[457] In this model, an attention-based gene expression encoder generates self-attention weights, a contextual attention layer ingests this gene embedding together with the SMILES encoding of a drug to compute an attention distribution over the SMILES tokens, in the genetic context. CNNs with variable kernel lengths were used to extract information about all possible substructures inside the SMILES sequence. The model outperformed others on a regression task involving prediction of drug IC50 values. Training was done using lenient splitting which prevented cell-drug pairs in the test data from being seen beforehand but did not prevent the model from observing how a given cell interacted with other drugs in the dataset and vice versa. The authors used drug sensitivity data from the publicly available Genomics of Drug Sensitivity in Cancer (GDSC) database for this study [458].

Morris et.al. 2020 proposed a transformer-based machine translation method to inform the segmentation of molecular substructures into binding/non-binding a target protein [459]. The authors translated SMILES encodings to IUPAC nomenclatures for a set of 83 million compounds from PubChem [460] database and used the resultant cross-representation attention embeddings as features to classify binding/non-binding compartments of molecules from BindingDB [446] to important proteins including HIV-1 protease.

12.3.4. Drug metabolism prediction

Metabolic processes in the human body can change a drug’s structure, diminishing its safety and efficacy. Therefore, investigation of the metabolic effect of a candidate drug is crucial in drug design studies. Litsa et al., 2020 fine-tuned a pretrained Molecular Transformer, and used an ensemble of them with beam search to find k-likeliest metabolites from every drug [461]. The Molecular Transformer [448] was pretrained on this dataset [462] consisting of 900,000 training instances. The network was further fine-tuned using a manually curated dataset combining samples from Drug-Bank (version 5.1.5) [441], Human Metabolome Database (HMDB) (version 4.0) [463], HumanCyc from MetaCyc (version 23.0) [464], Recon3D (version 3.01) [465], the biotransformation database (MetXBioDB) [466] and reaction rules from SyGMa [467]. Their network outperformed SOTA models including the BioTransformer [466].

13. Discussion

This paper presented an exhaustive summary of Transformer-based applications in healthcare for tasks such as clinical report generation, medical image segmentation and registration, molecular sequencing, drug-drug interactions, protein synthesis, surgical augmentation, and bio-physical signal analysis. Although relatively new, Transformers have become remarkably successful due to several reasons such as, parallelizable attention computation, ability to model long range dependencies, scalability, transfer learning, ability to produce contextual embeddings, interpretability and universal adaptability to various data modalities beyond text data. However, the parallelizable attention module at the heart of the Transformer network is computationally expensive and often needs to be optimized for efficient usage. In what follows, we highlight potential drawbacks of transformers, how to overcome them, and new directions enabled by Transformers.

13.1. Interpretability and explainability

Most deep learning systems are considered “black box” models because their inferences do not come with any discernable explanation. This lack of interpretability has traditionally prevented the systemic acceptance of AI-aided diagnostics in the medical domain. Transformers inherently provide some transparency through visualization of their attention weights. Trained attention weights elucidate contextual information significant for downstream inference. However, interpreting Transformers is challenging due to the frequent use of skip-connections and the dynamic nature of the model, which involves weight computation through matrix multiplication. Therefore, Transformer interpretability, albeit being an inherent property, is not trivial. Chefer et al. [468] show that Transformer attention is often fragmented and does not provide a robust explanation. They also proposed a novel way to compute relevancy based deep Taylor decomposition principle and propagate the scores through the transformer layers. In case of vision Transformers, Bohle et al. [469] proposed B-cos transformers, for holistic explanations for their decisions while retaining the performance to the baseline ViTs. Disease diagnosis prediction studies [470,471] have generated attention visualizations and cosine similarity between the learnt clinical diagnoses embeddings verified by expert clinicians to understand whether the trained model could capture the underlying semantic of diagnoses codes. However, there remains a need to develop novel techniques to improve the interpretability of Transformer models tailored towards healthcare AI.

13.2. Environmental impact

Advances in AI in recent years have come at the cost of a massive carbon footprint. Training a large-scale deep learning model is estimated to produce 626,000 lbs of carbon dioxide, equivalent to five automobiles’ lifetime emissions [472]. The number of computational resources researchers use to create SOTA models has doubled every three to four months [473]. Most emissions are associated with developing and training deep learning algorithms, whereas fine-tuning and adaptation contribute less [474]. Strubell et al. [472] suggested that researchers report hardware-independent training time measurements, such as the number of gigaflops required for training convergence and measuring model sensitivity to data and hyperparameters. The last decade has seen advancements in AI-augmented healthcare, on the one hand, and carbon emissions caused by AI systems that are detrimental to the climate and public health on the other. Large healthcare conglomerates and governmental agencies around the world should target net-zero carbon emissions. United Kingdom National Health Service has set a goal of net-zero emissions by 2040 [475]. Goals such as this are vital to promote the development of energy-efficient hardware and algorithms that make AI sustainable and globally accessible.

13.3. Computational costs

The reason behind the impact of Transformers is their high parametric complexity, flexibility to handle unequal input lengths and model scalability. However, Transformers’ ability to be trained on enormous datasets comes with expensive computational training budgets. The LLM GPT-3 [23] by OpenAI training is estimated to cost $4.6 million and 355 years of computing time using the Nvidia Tesla V100 device [476]. Google’s 530 billion parameters PaLM model is estimated to consume 103,500 KWh over 60 days [477]. Training and deploying large-scale AI models with high-end hardware requirements in healthcare settings is challenging. For example, for on-premise use in a hospital, a centralized compute cluster similar to ChatGPT might need to be maintained and interacted with using an API. However, healthcare settings typically need lightweight models to generate real-time predictions with minimal maintenance costs. Techniques for compressing deep learning models, such as pruning [478], knowledge distillation [479], and quantization [480], can be used to provide a more efficient model implementation for deployment within practical hardware constraints.

13.3.1. Model compression

Transformer models can be efficiently compressed by discarding some attention heads during the inference phase. Michel et al. [481] showed that models trained on multiple heads during training time need not require all the heads during test time. Similar redundancy has been observed in generating attention matrices from multiple heads [482].

13.3.2. Quantization

Quantization-based approaches reduce the number of bits/unique values required to represent model weights and intermediate layer activations. There has been growing interest among researchers in recent years in quantizing transformer networks. Shen et al. [483] observed ~2.3% degradation in performance with quantization down to 2 bits, corresponding to 13X compression of network parameters and 4X compression on embeddings and activations. It was observed that position embedding and the embedding layers are more sensitive to quantization than other operations.

13.3.3. Knowledge distillation

The knowledge distillation approach aims to train small networks (aka student) using the knowledge from the large model (teacher). Student models are obtained by reducing encoder width, number of heads, and number of encoders and replacing them with CNN, BiLSTM, or a combination [484]. Dimensional incompatibility between the student and teacher due to compact representations can be overcome by projecting teacher or student outputs [485]. Sun et al. [479] proposed patient knowledge distillation to compress large teacher BERT model trained on MIMIC-III dataset into shallow student models. Student models patiently learned from intermediate layers, which translated into improve performance and significant training-efficiency gain.

13.3.4. State space models

Transformer self-attention is capable of handling intricate interactions among sequence elements. However, this capability presents a limitation when applied to exceedingly long sequences, particularly in modalities like audio, video, and accelerometry where data extends continuously over time. State space sequence models [486], on the other hand, state space models excel in modeling long range sequences while maintaining computational efficiency. Conceptually, state space models can be seen as a fusion of recurrent neural networks and convolutional neural networks, offering linear or near-linear scalability to sequence length. A recent state space model named Mamba [487] has introduced a selection mechanism within its architecture, allowing it to make informed decisions about the information to propagate or discard based on its relevance to tokens in the sequence. Mamba leverages a hardware-efficient implementation inspired by FlashAttention [488], resulting in a remarkable 5X faster inference speed compared to Transformers. Mamba outperformed transformers of same size and matched the performance of transformers twice its size.

13.4. Fairness and bias

A model is biased when it exhibits undesired dependence on an attribute of the data that belongs to a specific demographic group [489], and could lead to unfair treatment of particular patient groups. Researchers have observed that bias often arises when the datasets used to train the models under-represent certain patient populations [490–492]. Although this is a prevalent bias problem during training, other sources of bias at all stages exist, including during problem formulation, data collection, data preprocessing, model development and validation, and model deployment (e.g., due to unmonitored drift) [493]. With the increasing scale of models and amount of data available, the existing biases and stereotypes perpetuate into the models leading to unfair and biased outcomes [49]. Thorough validation should be done before deploying the model to evaluate the performance of underrepresented groups. The models should be continuously monitored and audited for fairness and bias post-deployment.

13.5. AI alignment

The goal of AI alignment is even broader than preventing bias by striving to design AI systems that align with human values and goals. An AI system is considered aligned when the system behaves in ways beneficial to humans while minimizing the risk of unintended consequences and harmful outcomes [36]. LLMs sometimes confidently assert false claims that do not reflect facts, a phenomenon termed hallucination [494]. These hallucinations by the misaligned models fail to meet the user’s expectations of correct answers faithful to the existing sources. Ensuring AI systems are aligned with human values and goals is challenging because predicting and designing for every potential desired and undesired outcomes is difficult. As AI systems become more capable, they become increasingly susceptible to the alignment problem, which can result in unintended and harmful consequences [495]. AI alignment is especially critical in healthcare when deploying large-scale foundation models to ensure these models are ethical, responsible, respectful of patient privacy, and, most importantly, not causing harm. Healthcare professionals and the AI research community need to develop a clear set of standards and guidelines to establish ethical use of AI in health care.

13.6. Data privacy and data sharing

Preserving patient privacy is a required feature in all healthcare AI systems. Federal regulations based on the Health Insurance Portability and Accountability Act (HIPAA) regulate the development of AI models that use patient information [496,497]. Nonetheless, this also adversely impacts the development of large models such as Transformers that require large amounts of data. Utilizing data from a few sources, such as select public repositories, can skew the model inferences based on underlying limitations in dataset collection (different equipment, protocol, and cohort demographics), processing (specific heuristic or statistical preprocessing), and deployment (different metadata, availability, and maintenance). These biases can skew predictions that favor or adversely affect certain population groups over others, leading to a degradation in the quality and equity of healthcare for individuals from the protected group and stymieing the research on age, sex, or race-related medical conditions.

The Federated learning (FL) paradigm shown in Fig. 9 aims at developing a shared training model that can leverage data from multiple fragmented sources, such as different healthcare institutions, without divulging sensitive patient information [498]. FL communicates between various data sources by exchanging model-specific characteristics like parameters and gradients without exchanging patient information directly. Recent efforts in FL have targeted digital health objectives like determining patient clinical similarity [499,500], mortality and ICU length-of-stay [501], brain segmentation [502], and brain-tumor segmentation [503,504]. FL can perpetuate many healthcare innovations in the future. However, there are technical challenges in building an operational FL workflow, such as inhomogeneous data distributions, computational hardware differences, inconsistent privacy preservation settings, and resultant performance trade-offs [505].

Fig. 9.

Schematic of Federated learning with a central server that interacts with training nodes at different locations continuously updating the model parameters without exchanging the data between local and central servers.

14. Conclusion

Transformer models have demonstrated enormous potential in a wide variety of healthcare applications. They possess a unique ability to model various data modalities, including images, clinical text, bio-physical signals, structured EHR, social media and genomic data. From disease diagnosis to drug discovery, Transformer models exhibit the potential to improve patient outcomes and advance medical research. However, various challenges and limitations remain to be addressed before they are widely accepted into regular clinical practice. These include data limitations, biases, privacy, security, and truthfulness. The majority of the models currently in use are task-specific, and there is a need to utilize robust multimodal inputs in many cases. Nevertheless, the future of AI in healthcare is optimistic, with promising advancements and opportunities presented by large-scale transformer models.