Large language model to multimodal large language model: A journey to shape the biological macromolecules to biological sciences and medicine

Abstract

After ChatGPT was released, large language models (LLMs) became more popular. Academicians use ChatGPT or LLM models for different purposes, and the use of ChatGPT or LLM is increasing from medical science to diversified areas. Recently, the multimodal LLM (MLLM) has also become popular. Therefore, we comprehensively illustrate the LLM and MLLM models for a complete understanding. We also aim for simple and extended reviews of LLMs and MLLMs for a broad category of readers, such as researchers, students in diversified fields, and other academicians. The review article illustrates the LLM and MLLM models, their working principles, and their applications in diversified fields. First, we demonstrate the technical concept of LLMs, working principle, Black Box, and the evolution of LLMs. To explain the working principle, we discuss the tokenization process, token representation, and token relationships. We also extensively demonstrate the application of LLMs in biological macromolecules, medical science, biological science, and other areas. We illustrate the multimodal applications of LLMs or MLLMs. Finally, we illustrate the limitations, challenges, and future prospects of LLMs. The review acts as a booster dose for clinicians, a primer for molecular biologists, and a catalyst for scientists, and also benefits diversified academicians.

Graphical abstract

Chakraborty and colleagues extensively illustrate LMMs and MLLMs for a better understanding, along with the working principle of an LLM with the tokenization process, token representation, and token relationships. Moreover, the diversified applications of LLMs and MLLMs are discussed in biological macromolecules, biological sciences, medical science, and other areas.

Introduction

Since the launch of ChatGPT by OpenAI on November 30, 2022, large language models (LLMs) have become popular quickly. The medical and scientific communities have been thrilled to use LLMs in various biology, medicine, and science areas. During the last phase of 2022, Stokel-Walker reported in Nature that ChatGPT can write Smart essays.¹ After that, people found that the written responses of this LLM chatbot were rapid and often invariable from humans. Conversely, researchers found that LLMs can process the text from queries and can respond, revolutionizing the field.

LLMs are one of the significant achievements among the recent noteworthy technologies of artificial intelligence (AI). The rapid development of AI has led to the development of this sophisticated technology of LLMs. As a component of AI, LLMs are trained with billions of words derived from internet-based content, books, articles, and massive texts. During the production of text, the LLM chatbots also extract input from billions of unidentified general web pages. However, using AI and NLP (natural language processing), LLM chatbots can recognize questions and offer automated answers. They are LLM-based dialog agents.^2,3 This model can answer free-text queries without being trained with a specific topic. The model’s explicit training objective is constructed on the generative pre-trained transformer (GPT) architecture. It is to predict the next word efficiently and can be prepared in a sentence or paragraph. Their performance has become increasingly human-like through dialog as a dialog agent (Figure 1). The most essential benefit of GPT models is their processing speed. The GPT model can answer complex input queries in just a few seconds. LLMs use a machine learning model based on the architectures of neural networks.^2,4,5,6 The skill resembles cognitive capability.

Figure 1.

The working principle of a LLM in generating a single-token sequence

The model generates tokens, ultimately forming a complete sentence.

The publicly popularized LLM chatbot, ChatGPT was GPT model 3.5. Millions of users started to use it within a few months after it was released. It was noted that about 100 million users used ChatGPT within 2 months after its launch.⁷ Subsequently, interest in LLM chatbots has increased very fast across academic domains. Afterward, it has been used across industrial domains.^8,9 In medical science, it has been applied in different areas such as cardiology,¹⁰ orthopedics,¹¹ radiology,¹² infectious disease,¹³ drug resistance,¹⁴ surgery,¹⁵ etc. Other than medical science, ChatGPT has been used in different academic areas such as pharmacology and drug discovery,¹⁶ law,¹⁷ education,¹⁸ biomedical engineering,¹⁹ finance,²⁰ etc.

Subsequently, in 2023, OpenAI released GPT model 4, which is a multimodal LLM (MLLM). MLLMs can be trained with video, audio, or image, along with all the training parameters that are used to trained LLM. It can be trained with more comprehensive training parameters. MLLM is a robust model.²¹ Therefore, it is considered a more advanced version of LLM. GPT-3.5 and GPT-4 used different parameters and tokens. It was reported that GPT-3 was using over 175 billion machine learning parameters. GPT-3’s parameters (175 billion) help the LLM’s vast understanding of language and knowledge across various domains. This machine learning model was trained on 300 billion tokens. GPT-3.5 is more fine-tuned than GPT-3’s abilities. GPT-3.5 is more refined and exhibits more accuracy compared with GPT-3. Similarly, the GPT-4 model was developed to incorporate about 1.8 trillion parameters. At the same time, this MLLM is trained with 13 trillion tokens.²²

Here, this comprehensive review article illustrates the basic technical concept of LLM, its working principle, LLM and its Black Box, and the evolution of LLMs. We also illustrate the application of LLMs in medical science, biological science, and other areas. We illustrate the multimodal applications of LLMs or MLLMs. Finally, we illustrate the limitations, challenges, and future prospects of LLMs. The article has been simplified and extended, which will help a broad group of readers to understand the topic better.

Basic technical concept of LLMs

The role of an LLM is to respond to questions. The model is trained with a considerable number of tokens. It is based on transformer architecture and NLP. The transformer architecture is a type of deep learning neural network architecture. The significant questions are: Which token will most probably come next? What will the weightage of a token be? The LLM works through the probability distribution. The probability distribution is as follows:

P (ꞷ_n+1|ꞷ₁ … …ꞷ_n), where ꞷ₁ … ꞷ_n is a sequence of tokens (the context) and ꞷ_n+1 is the predicted next token.²

How the LLM works

Language models (LMs) have been developing for several years to enhance the intellect of machine languages. Technically, LMs function with probability value allocation to word sequence and deliver appropriate text output (Figure 1).²³ LMs utilize tokens as basic units for understanding and generating text output, where tokens refer to parts of text that can be words, characters, or sub-words. With the use of tokens, LMs can grasp the connection and relation among words, enabling them to produce grammatically textual output that is right.²⁴ LLMs, i.e., pre-trained LMs with millions of parameters, can grasp and produce output strikingly similar to human language. The LLMs operate with the tokenization theory and can apply millions of tokens to produce output.

Tokenization process

In tokenization, the text is split into small sections known as tokens, which are subsequently utilized for analysis in LLMs. Tokenization is crucial in text pre-processing, preparing input tokens for LMs. The commonly employed methods for tokenization are WordPiece and BPE (byte-pair encoding), which are utilized by significant models such as bidirectional encoder representations from transformers (BERT) as well as the GPT. Nevertheless, the tokenization’s effect may vary in languages that are rich morphologically, such as the Turkic languages, where the addition of prefixes and suffixes can generate numerous words. In such cases, a newly formed tokenizer operating at the morphological level can challenge the already established tokenizers.²⁵ Within LLMs, tokenization also holds importance in speech processing. The tokenization process in discrete speech notably contributes to integrating speeches into the LLMs.

Nonetheless, the discrete process gives rise to information loss, which impairs the performance. One unique speech-representing codec called RepCodec has been proposed for semantic speech tokenization to enhance the functioning of these speech tokens (discrete).²⁶ In the context of text summarization, tokenization is an essential technique for obtaining the needed information precisely.²⁷ Summaries help lower the time needed to read and facilitate researching documents. However, tokenization can potentially introduce ambiguity, and there exists a lack of clarity on whether the divided tokens attain optimal performance for the intended task.²⁸ An innovative, straightforward, and effective method named GrowLength has been introduced to expedite the pre-training process of LLMs. This method gradually enhances training length during the pre-training period, thus reducing computing expenses and improving efficiency.²⁹

Token representation

LLMs utilize embedding and encoding mechanisms to represent tokens. These mechanisms enable the model to comprehend and generate natural language by capturing the tokens' semantic and syntactic information. Embedding mechanisms map tokens to high-dimensional vectors, thereby facilitating the model’s grasp of their semantic meaning. One commonly adopted approach involves utilizing pre-trained embeddings of the words, such as GloVe or Word2Vec, which offer distributed descriptions of words depending on their contextual usage.³⁰ Within LLMs, token embeddings are acquired through pertaining and fine-tuning, enabling the model to capture intricate linguistic patterns and relationships between tokens.³¹ Encoding mechanisms, on the other hand, process token embeddings to capture sequential and contextual information. In LLMs, this is typically accomplished by employing self-attention mechanisms such as the transformer architecture, which allows it to allocate important weights for each token in the overall input sequence.³² Consequently, the model becomes proficient in capturing long-range dependencies and contextual information, which proves critical for numerous NLP tasks. In addition to self-attention, recent research has explored novel token encoding mechanisms, such as recurrent alignment and contrastive losses, designed to encapsulate nuanced semantic relationships between tokens and optimize the embedding space.³³

Token context and relationships

LLMs such as GPT-3 and BERT can capture context and establish relationships between tokens by generating token representations that retain crucial contextual knowledge necessary for various tasks. An exemplification of this is observed in the SPAE (Semantic Pyramid AutoEncoder), which empowers LLMs that are frozen to undertake comprehension and generate tasks requiring non-linguistic procedures such as videos or visual representation of objects. SPAE accomplishes this by converting raw pixels into interpretable lexical tokens taken out from the word stock of the LLM, thereby capturing both semantic meaning and intricate details necessary for visual reconstruction.³⁴ In addition, contextual LMs, such as BERT produce token representations that retain context-specific knowledge essential for tasks at the type level, thereby indicating the context sensitivity of processes such as similarity estimation and relatedness estimation.³⁵ The contextual knowledge facilitates LLMs in capturing intricate relationships between tokens, rendering them suitable for various tasks, including image generation and semantic estimation. Attention mechanisms in the models play a pivotal role in contextual understanding by allowing the models to selectively focus on different segments of the input sequence to capture pertinent information. An illustration of this is the tri-attention framework in NLP, which explicitly incorporates the query, key, and context interactions by including context as the third dimension in the computation of relevance scores. Thus, this framework surpasses traditional bi-attention approaches and pre-trained neural LMs in various NLP jobs³⁶ Furthermore, the Vit-BiGRU-Attention sentiment classification model utilizes attention mechanisms to assign varying weights to individual words, thereby enhancing the comprehension of emotions and determining the polarity of emotions in user comments, ultimately leading to improved accuracy in sentiment classification.³⁷

As a machine learning model, LLM architecture primarily consists of numerous layers of neural networks, such as recurrent, feedforward, embedding, and attention layers. It uses a probabilistic model, tokenization process and representation, and neural architecture at a time to generate human-like language text.

LLM and its “Black Box”

Neuroscience models mainly guide artificial neural networks (ANNs), which brain mechanisms encourage and develop. Unfortunately, networks generated by neurons are described as unclear as those generated by the brain. The data was observed to diffused, so it was not straightforward to solve. AI is developed using the ANN model, which cannot be adequately described in the Black Box of AI. Researchers are trying to describe the inner workings of a complex Black Box model of AI.³⁸ The Black Box can be applicable to LLM or ChatGPT models.^39,40 However, the interpretability of the LLM output cannot be adequately explained. At the same time, tokenization processes cannot be adequately explained. Therefore, the output of LLMs can be described as the Black Box of LLMs and understanding it is a challenge for the researcher.^41,42,43

Evolution of LLMs

The evolution of LLMs is evidence of the rapid pace of AI research and innovation. The journey of LLMs started with simpler LMs, and the present journey continues with the development of massive neural networks such as GPT-3.5 and GPT-4 (Figure 2). The present LLM, GPT-4, is a multimodal LLM with immense capabilities across vision, video, audio, language, and 3D. It also claims billions of parameters along with a safety research and monitoring system.⁴⁴ The timelines of LLM evolution include multiple stages from the last few years. It evolved significantly over the years, with advancements in model architecture and training methodologies.⁴⁵

Figure 2.

The evolution of a LLM through chatbots

(A) The evolution of LLM chatbots with a timeline that finally developed the recent MLLM. (B) Some significant medical chatbots follow the LLM or MLLM models.

Before the era of deep learning, rule based and statistical methods were dominated by NLP. Models such as ALICE (1995) and Eliza (1966) put the foundation for conversational agents. However, they were rule based and lacked a proper language understanding.⁴⁶ In the 2000s, statistical models such as Hidden Markov Models and n-grams improved language processing by considering probabilities of word sequences. These models were data driven but limited in handling complex language distinctions.⁴⁷

In subsequent consideration of advancement LLMs, a recurrent neural network (RNN) model is an ANN model that processes and converts sequential data inputs into sequential data outputs. RNNs were among the first neural network architectures applied to sequential data, such as text, and were initially created in 1980s. Currently, it is ideally suited for machine learning problems involving sequential data. However, it suffered from vanishing gradient problems, limiting it ability to capture long-range dependencies. The long short-term memory (LSTM) networks addressed the vanishing gradient problem by introducing a memory cell in 1997.⁴⁸ It allowed them to capture long-term dependencies in sequential data, improving performance in language-related tasks.⁴⁹ Then, in 2013, word embeddings, such as Word2Vec and GloVe, represented words as continuous vector spaces. These embeddings captured semantic relationships between words, offering better representations than traditional methods.⁵⁰ In 2017, statistical language processing came out, and the introduction of deep learning techniques revolutionized NLP. The RNN and LSTM networks showed promise in sequential data tasks, paving the way for more advanced models. The appearance of Google transformer architectures and attention mechanisms marked a turning point in LLM evolution. Transformers use attention mechanisms to process words about all other words in a sentence, significantly improving contextual understanding.⁵¹

OpenAI’s GPT series, starting with GPT-1, then GPT-2, and the massive GPT-3, showcased the power of pre-trained models fine-tuned for various tasks. GPT-3’s unprecedented scale generated coherent and contextually relevant text.^52,53 GPT-2 was released in 2018 and can use 1.5 billion parameters. This version of GPT used the extensive collection of free novel books dataset called the BooksCorpus dataset, containing 11,308 novels. It has been noted that it includes about 1 × 10⁹ words or around 74 million sentences.⁶ Similarly, GPT-3 was released in 2020, and it can handle 175 billion parameters. GPT-3 is assumed to be 100 times more extensive than the previous GPT (GPT-2). The training dataset comprises 45 terabytes and 5 corpora. It contains Wikipedia, WebText2, Common Crawl (webpages), Books1, and Books2. GPT-3 is one of the most sophisticated LLMs until today.⁶ Due to user requirements, GPT-3 was evolved into GPT-4, an MLLM.

Simultaneously, several other pre-trained LMs were developed in recent years (Table 1), which include open pre-trained transformer, pathways language model (PaLM), anthropic-LM, language model for dialog applications, MT-NLG, and LLaMA.^6,54

Table 1.

Different recently developed LLMs and their developers

Sl. no.	LLM name	Developer	Year of release	Remarks
1.	LLaMA	Meta	2023	It is the collection of foundation language models ranging from 7B to 65B parameters. This is modeled on trillions of tokens, and shows that it has the potential to train state-of-the-art models using publicly obtainable datasets exclusively, deprived of resorting to exclusive and inaccessible datasets.
2.	ChatGPT	OpenAI	2022	It is based on an LLM, and enables users to refine and steer a specific conversation toward a preferred length, style, format, and level of detail, as well as language.
3.	Flamingo	DeepMind	2022	It is a single visual language model (VLM), sets a new state-of-the-art in few-shot learning on an extensive range of open-ended multimodal tasks. It is also able to tackle a number of challenging problems with just a handful of task-specific examples, lacking any additional training prerequisite.
4.	DALL-E	OpenAI	2021	It uses developed text-to-image models by OpenAI by deep learning methods to produce digital images from natural language descriptions, known as prompts.
5.	Anthropic-LM	Anthropic	2023	A combination of AI chatbot and LLM that can help doctors to diagnose diseases accurately and efficiently. An improved version has also been designed to be safer than other models and is sometimes called a potential ChatGPT killer.
6.	Turing-NLG	Microsoft	2020	It is a 17-billion-parameter language model by Microsoft that outperforms the state-of-the-art on many downstream NLP tasks. In addition to completing an unfinished sentence, it can generate direct answers to questions and summaries of input documents.
7.	Minerva	Google	2022	Minerva is based on the pathways language model (PaLM). It is a 540-billion-parameter language model that can generalize across different domains and tasks. It was trained on a dataset of scientific papers and web pages that contain mathematical expressions.
8.	Wu Dao 2.0	Academia	2021	It has the ability to perform natural language processing and image recognition, in addition to generation of text and images. The model cannot only write essays, poems, and couplets in traditional Chinese, it can both generate alt text based on a static image and generate nearly photorealistic images based on natural language descriptions.
9.	Imagen	Google	2023	Imagen is a text-to-image diffusion model with an exceptional degree of photorealism and a deep level of language understanding. It builds on the power of large transformer language models in appreciative text and hinges on the strength of diffusion models in a high-fidelity image generation group.
10.	GPT-3	OpenAI	2020	It is a decoder-only transformer model of a deep neural network, which overtakes recurrence and convolution-based architectures It contains sustainable identified themes, emotions, and sentiment from surveys, reviews, live chat logs, help desk tickets, and more.
11.	Megatron	Nvidia	2021	It is an extremely optimized and well-organized library for training LLMs. With this model parallelism, language models can be trained with billions of weights and then used in NeMo for downstream tasks.

LLMs stand as a superior consequence of the remarkable progress in AI research. Their evolution from basic LMs to powerful transformers has redefined the possibilities of NLP. While their applications across domains offer immense benefits, ethical considerations must guide their deployment to ensure a responsible and equitable social integration.⁵⁵ As we continue to explore the potential of LLMs, a balanced approach that combines technological advancement with ethical mindfulness will shape the future of AI and human-machine interactions.⁵⁶ A continuous drive marks the evolution of LLMs for larger models and further evolution into MLLM, which provides better pre-training strategies and increased attention for future considerations to ensure responsible AI development and deployment.

Prompt engineering

LLMs or chatbot technologies use deep learning and NLP to learn the language patterns for conversations with humans from a large amount of text data. In conversations with humans, LLMs depend on proper input or high-quality prompting, which is called "prompt engineering."^57,58 Therefore, asking the right question to an LLM is essential. This is a new area of research that focuses on refining, designing, and implementing instructions or prompts to improve LLMs' output.

Different scientists use prompt engineering to yield more valuable outputs for precise instructions from LLMs. Kleinig et al. illustrated how to use prompt engineering in ophthalmology.⁵⁷ Venerito et al. explained the use of prompt engineering in rheumatology research and described how prompts are methodically constructed in this area of research.⁵⁹ Polak and Morgan explained how to extract proper data from research papers.⁶⁰ Therefore, prompt engineering is valuable in biological macromolecules, biological sciences, and medicine.

Application of LLMs in biological macromolecules

Recently, LLMs have been used in various fields of biological macromolecules (Figure 3). Researchers have been trying to use LLMs to understand the properties and functions of biological macromolecules.

Figure 3.

Some significant applications of a LLM in the field of biological macromolecules.

Proteins

LLMs have been used in various fields of protein research. Researchers have been using LLMs to understand the properties and functions of proteins. Therefore, researchers are using LLMs in different areas of protein science. They have also attempted to develop protein-centric LLMs to perform protein-related tasks. Recently, Zhuo et al. have suggested one LLM for protein entitled PROTLLM to perform protein-language-related and protein-associated tasks. It is a versatile crossmodal LLM that can perform dynamic protein-centric assignments and dynamic protein mounting.⁶¹ Likewise, Guo et al. suggested one LLM for the protein named Proteinchat to perform chatbot-like functionalities on three-dimensional protein structures.⁶² Similarly, Wang et al. suggested another LLM for the protein named ProtChatGPT for comprehending proteins with an LLM.⁶³ At the same time, Wang et al. developed another LLM model for a protein called InstructProtein. The LLM model helps to comprehend the protein language and align humans through knowledge instruction.⁶⁴ However, LLMs have started to be used to explore novel research of different areas of protein structure and function, and other different areas.

Nucleic acid research

LLMs have recently been used in various fields of nucleic acid research. We recently explained LLM’s role in the field.⁶⁵ Researchers have conducted GeneTuring using six GPT models: new Bing, ChatGPT, BioMedLM, BioGPT, GPT-3, and GPT-2. The open company developed ChatGPT, GPT-3, and GPT-2. The GeneTuring test was conducted with an exhaustive QA database with 600 genomics questions. New Bing’s overall performance was the best.⁶⁶ Ji et al. use a pre-trained transformer model called DNABERT. Using this pre-trained model, they have indicated transcription factor binding sites, splice sites, and promoters.⁶⁷ Recently, a group of researchers used DeepMind to forecast the effective and improved gene expression prediction from DNA sequences.⁶⁸

Polysaccharides and lignin

LLMs have recently started to be used to explore various fields of polysaccharides and lignin. Researchers have studied ChatGPT’s performance in responding to glycobiology and carbohydrate chemistry queries. Williams and Fadda explored ChatGPT’s answer style to different glycobiology and carbohydrate chemistry questions. They found that the model can answer short and descriptive questions correctly. However, they found that answers contained fabricated text.⁶⁹ Researchers developed an LLM for material modeling. Buehler developed MechGPT for materials modeling and mechanics, which can be used for lignin modeling.⁷⁰ David et al. developed an LLM or ANN to assess sugar output from Kraft waste-based lignocellulosic pre-treatments. They tried to understand that domain-specific knowledge can help accelerate the progression of lignocellulosic waste pre-treatment.⁷¹

Application of LLMs in biological sciences

Drug discovery and development

LLMs play a significant role in drug discovery, offering valuable contributions in various stages of the drug development process. They can compute extensive pieces of scientific literature, extracting relevant information about potential drug targets, biomarkers, and mechanisms of action.⁷² Likewise, they aid in identifying potential drug targets by analyzing biological and biomedical texts to understand the relationships between genes, proteins, and diseases. LLMs can predict potential drug interactions, assessing the likelihood of adverse or synergistic therapeutic effects.⁹ Moreover, the LLMs can contribute to monitoring adverse events related to drugs by analyzing medical literature, clinical trial reports, and social media. They also help match eligible patients to clinical trials by analyzing electronic health records, medical literature, and patient data. The application of LLMs in drug discovery demonstrates their ability to process and understand large volumes of biomedical information, offering valuable insights that can identify novel therapeutic targets, which initiate faster methods of novel therapeutics development.¹⁶ The application of LLMs in drug discovery demonstrates their ability to process and understand large volumes of biomedical information, offering valuable insights that can trigger the faster process of identification of novel therapeutics and their development.

Molecular biology and computational biology

Chatbot technology is evolving very fast and is appearing as a new AI tool for molecular biologists and computational biologists. Several researchers have checked the potentiality of LLMs in molecular biology. Recently, Ross and Gopinath illustrated the process of learning the structural biophysics of DNA using an LLM.⁷³ Lubiana et al. give ten tips to assist computational biologists in optimizing the research workflow with an LLM or ChatGPT. The tips include enhancing data clean up, writing code efficiently, improving data visualization, and prompt engineering. They also advise that we should only depend a little on ChatGPT.⁷⁴ Tiwari et al. used ChatGPT/GPT-4 or an MLLM to understand pathway enrichment and annotation gaps with comparative analysis using the manual curation process (conventional process). They determined some promising capabilities of this MLLM.⁷⁵ Levine et al. developed Cell2sentence, a GPT-2 model based on an LLM. It can be used to teach biological science, especially single-cell transcriptomics.⁷⁶ However, the application of LLMs to MLLMs is increasing day by day in molecular biology and computational biology.

Application of LLMs in medical sciences

LLM models have been used from time to time and have been applied in different fields of medical science (Figure 4). LLMs can enhance diagnosis and support clinical judgment in medicine. However, to make them function well in the medical field, particular difficulties must be overcome.⁷⁷ LLMs may completely transform the healthcare industry by improving diagnosis accuracy, predicting the course of diseases, and supporting physicians in their decision-making.^78,79

Figure 4.

Some significant clinical applications of a LLM in the medical field.

LLMs in medical education

LLMs might be improved by concentrating on specialized medical literature to stay relevant and up to date. They can also be customized for different languages and scenarios, improving global access to medical knowledge and information.⁷⁷ Recently, there have been numerous instances where the application of LLM technology, notably ChatGPT, has been documented.⁶ After passing the US Medical Licensing Exams, ChatGPT became well-known in the medical community. GPT-4 performs far better than GPT-3.5, its predecessor.^80,81 These medical LMs, however, were exceptionally trained on texts related to medicine or biology. They come in useful for jobs such as question-answering, translating, and summarizing. Examining if smaller models trained on pertinent data can perform as well at a reduced cost is necessary, given the high cost of training and utilizing these models. For example, at the cost of $600, the Center for Research on Foundation Models at Stanford University made a model named Alpaca that matched the performance of OpenAI’s text-davinci-003 with just 4% of its parameters.⁸² By strengthening critical medical competencies such as factual knowledge and interpersonal communication, LLMs can elevate the standard of care for patients. For example, ChatGPT has demonstrated success in medical licensure exams demonstrating its substantial medical knowledge and ability to participate in medical reasoning.^80,81,83 LLMs' medical reasoning and concept understanding can be improved even further by providing focused instruction that includes questions akin to those on a medical test and expertly chosen sample answers. GPT-4 presently exhibits the most significant medical domain knowledge among LLMs. Nonetheless, LLMs face a fundamental limitation: they frequently reproduce pre-existing medical biases⁸⁴ and sustain inequalities associated with socioeconomic status, gender, ethnicity, and other characteristics.^80,85

The use of LLMs in healthcare is progressing rapidly, driven by the widespread availability of LLMs, including their availability to students and some research-based initiatives. This involvement can be validated by the involvement of ChatGPT in the Epic Systems Corporation’s software, as reported in a recent article.⁸⁶ The potential applications are diverse, ranging from streamlining administrative tasks such as assisting on the instructions related to patient discharge, insurance filings, as well as obtaining some prior authorizations for medical services.⁸⁷ Moreover, there is a prospect of enhancing the standard-of-care by finding the older medical history from intricate records of patients along with a detailed checking on some of the standardized operating procedures. Among the emerging applications, two stand out: the capability of LLMs to analyze vast amounts of data in an unstructured form in electronic health records and their potential to aid in clinical documentation.^87,88 Incorporating these models into the educational framework can stimulate deep critical thinking, encourage creative work, and provide innovative learning experiences. Furthermore, gaining a profound understanding of these models prepares the students for working in the healthcare industry, which is also closely related to AI. Evaluating the application of ChatGPT in medical science is a crucial stride in harnessing the technological potential to guide forthcoming changes in the new era of medical science. More notably, the next wave of healthcare professionals needs to not only be familiar with these modern technologies but also possess the skills to responsibly and effectively employ them in the delivery of patient care.⁸⁹ In addition, the emergence of ChatGPT has generated new insights into AI-powered chatbots and their possible uses, attracting considerable attention worldwide. In recent months, there has been growing interest among scientists and medical professionals in implementing the applications of LLMs in medicine.⁹⁰

LLM-based medical tool or device for medical education and research

LLMs are trained on medical data based on various codes and text. After examining this training set of data for more than 80 medical LMs, Wornow et al. distinguished two significant groups.⁹¹ Firstly, textual resources such as progress notes or PubMed abstracts train specific models. They learn by making predictions about the words that will come next in these papers, just like generic LMs such as GPT-3 do. The effectiveness of utilizing domain adaptation, transfer learning, and alternative methodologies in the medical field is demonstrated by multiple examples of LLMs that have been specifically fine-tuned for medical purposes. BioBERT, a biological LM based on the BERT architecture, was refined by leveraging large biomedical datasets such as PMC full-text articles and the abstracts available in PubMed. As a result, there were significant improvements in several biological NLP tasks, such as problem-solving, relation extraction, question-answering, and named entity recognition.⁹² ClinicalBERT, a distinct model, was subjected to fine-tuning using the MIMIC-III dataset, which comprises the electronic health records from patients in critical care units. Fine-tuning exhibited enhanced efficacy in clinical NLP assignments, including diagnosis categorization, patient mortality rate prediction, and de-identification.⁹³ BlueBERT, a model constructed according to the basis of the BERT architecture, has already been pre-trained on an extensive collection of biomedical texts and has demonstrated exceptional performance in multiple biomedical NLP tasks. These tasks include named relation extraction, biomedical problem-solving, and entity recognition.⁹⁴ The cases above highlight the effectiveness of utilizing domain-specific fine-tuning, transfer learning, domain adaptation, and alternative methods to harness the capabilities of LLMs in various fields of medical science.⁷⁷ Recently, a specialized version named Med-PaLM 2 (Google) that was trained on medical data achieved state-of-the-art results similar to the level of proficiency exhibited by human doctors.⁹⁵ Recently, specialized LLMs in different fields of medical science have been developed and applied periodically, and some of them are PMC-LLaMA, ClinicalCamel, MedAlpaca, BioGPT, BioMedLM, Med-PaLM2, and ChatDoctor (Table 2). These specialized LLMs have revolutionized the field of medical science.

Table 2.

Application of specialized LLMs in different fields of medical science

Sl. no.	LLM	Year of release	Remarks	Reference
1.	PMC-LLaMA	2023	PMC-LLaMA, an open-source language model that is developed by refinement of an open-source language model on a total of 4.8 million biomedical academic papers for added injecting medical knowledge, improving its capability in the medical domain.	Wu et al.96
2.	ClinicalCamel	2023	It is an open LLM obviously tailored for clinical research. Fine-tuned from LLaMA-2 using QLoRA, Clinical Camel achieves state-of-the-art performance across medical benchmarks among openly available medical LLMs. Leveraging efficient single-GPU training.	Toma et al.97
3.	MedAlpaca	2023	MedAlpaca was developed by instruction fine-tuning of the LLaMA 13B and 7B models on Medical Meadow data. It is also the assembly of reformatted instruction-response pairs with datasets for medical NLP tasks and data derived from various internet sources.	Han et al.98
4.	BioGPT	2023	BioGPT is a domain-specific GPT language model for biomedical text generation and mining. BioGPT follows the transformer language model backbone, and is pre-trained on 15M PubMed abstracts from scratch.	Luo et al.99
5.	BioMedLM	2022	BioMedLM is based on a HuggingFace GPT model (decoder-only transformer) with 2.7B parameters and a maximum context length of 1,024 tokens. It also uses a custom biomedical tokenizer trained on PubMed abstracts with a vocabulary size of 28,896.	Karkera et al.100
6.	Med-PaLM2	2022	Med-PaLM is a large language model (LLM) designed to provide high quality answers to medical questions. It is also available to Google Cloud customers, who are able to explore a range of applications, from basic tasks to complex workflows. It has been aligned to the medical domain and evaluated using medical exams, medical research, and consumer queries.	Luo et al.101
7.	ChatDoctor	2023	This is a specified language model with improved accuracy in medical advice refining the large language model meta-AI (LLaMA) by a large dataset of patient-doctor dialogs obtained from a widely used online medical consultation platform.	Li et al.102

LLMs in different clinical fields

By examining extensive medical data, the LLMs can quickly develop specialized knowledge in various medical sectors, including radiology, pathology, and oncology.^103,104,105 Notably, the release of OpenAI’s ChatGPT quickly sparked a massive revolution in other clinical fields, such as ophthalmology, nephrology, cardiology, and orthopedics.¹¹ Some LLMs are also trained using patient record sequences of medical codes. These models pick up new information by anticipating the codes for the next day or comprehending the time intervals between particular codes. They consider the sequence and the chronology of medical occurrences documented in a patient’s file. For instance, if trained on some particular codes, these models can predict the chance of a stroke, heart attack, or renal failure. Rather than producing text, these models yield a fixed-length, high-dimensional vector that machines can read as an “embedding” of the patient’s medical record. With as little as 100 training data examples, these embeddings can be used to build models predicting 30-day readmissions, prolonged hospital stays, and in-patient death.¹⁰⁶ Domain-specific LLMs tailored to specific domains could offer valuable new features in various clinical domains. For instance, foresight, an LLM that was built on the GPT architecture and trained on unstructured data from over 811,336 electronic health records, showed promise in accurately predicting and forecasting outcomes during validation trials.¹⁰⁷

However, education and specific training are essential for effectively integrating LLMs into medical practice. Given the growing importance of LLMs in healthcare, medical personnel must fully understand their capabilities and limitations. This knowledge will allow them to utilize these technologies in clinical settings effectively. To fully equip future medical practitioners, medical curricula must incorporate the fundamental principles of utilizing LLMs. It will ensure that students gain the necessary knowledge and abilities to navigate and exploit these technological developments.⁷⁷

Application of LLMs in other areas

Financial modeling and sentiment analysis

LLMs are increasingly used in financial modeling and sentiment analysis, offering advanced NLP capabilities to analyze and interpret financial data and model and measure market sentiment. It can transcribe and analyze earnings calls, extracting critical information about a company’s performance, outlook, and management discussions to know the market trends.¹⁰⁸ In addition, LLMs can analyze text data related to companies, industries, and economic conditions to assess and quantify various financial, operational, and market risks.¹⁰⁹ Subsequently, LLMs process customer reviews, feedback, and comments to measure sentiment about products, services, or brands. This information is valuable for companies to understand customer satisfaction, allowing companies to make data-driven decisions to improve strategies in both financial modeling and sentiment analysis.¹¹⁰ LLMs influence their ability to understand and generate human-like language to process vast amounts of textual data, providing valuable insights for decision-making in the financial domain.¹¹¹

Legal research and analysis

LLMs are increasingly employed in legal research and analysis, transforming how legal professionals access, process, and understand legal information. They assists legal researchers in analyzing and summarizing case law, providing concise overviews of legal precedents and decisions; more specifically, they automatically summarize lengthy legal documents, including contracts, pleadings, and briefs, facilitating quick review by legal professionals.¹¹² LLMs contribute to trademark searches and analysis by processing and summarizing relevant information from trademark databases and legal texts. Applying LLMs in legal research and analysis enhances efficiency, accuracy, and the accessibility of legal information, transforming how legal professionals approach various tasks within the legal domain.¹¹³ LLMs can review and analyze legal contracts, helping legal professionals identify vital terms, risks, and obligations.

Customer service by chatbots and email response

LLMs power chatbots for customer support, answering queries, resolving issues, and conversationally providing information. LLMs enhance the capability of chatbots to handle multiturn conversations, maintaining context and providing coherent responses across different user inputs. They engage in natural language conversations, accurately interpret user queries, and provide more relevant responses, providing users with a more human-like and intuitive interaction experience.^113,114 They also can help analyze user sentiment during interactions, allowing chatbots to respond appropriately and adapt their tone based on the user’s emotional context.¹¹⁵ LLMs can help generate responses to customer emails, improving efficiency in handling customer inquiries. This helps to categorize and prioritize incoming emails, directing them to the appropriate department or team for more efficient handling in both chatbot-driven customer service and email communication. LLMs are crucial in automating processes, improving response accuracy, and enhancing the overall customer experience by providing more natural and intelligent interactions.¹¹⁶

Education field

LLMs have found diverse applications in the education sector, transforming various aspects of teaching, learning, and administrative processes, supporting the automated grading of assignments and exams, and providing quick and consistent feedback to students. Furthermore, they can assist in language learning by providing grammar explanations, vocabulary explanations, and conversational practice.¹¹⁷ Different conditions provide reliable assistance to students with homework, offering explanations and guidance on various subjects. LLMs help generate research proposals and offer guidance on structuring and framing research questions. Presently, the LLMs contribute to language translation, breaking down language barriers and making educational content accessible to a global audience. Finally, they contribute to student performance data analysis, helping educators make data-driven decisions.¹¹⁸ Therefore, the application of LLMs in education is vast and continually evolving, potentially enhancing learning experiences, streamlining administrative processes, and providing personalized support to students and educators.¹¹⁹

Marketing field

LLMs play a crucial role in marketing across various aspects, applying NLP capabilities to enhance communication, analyze data, and optimize strategies. It is capable in creating compelling copy for digital advertising campaigns, ensuring messages resonate with the target audience, blog posts, articles, and other content for marketing purposes, and maintaining consistency and quality.^120,121 Subsequently, the LLMs can contribute in analyzing competitor strategies, monitoring industry trends, and identifying areas for differentiation. Likewise, creating personalized marketing messages based on user data improves customer engagement and conversion rates.¹²² Besides, the LLMs power chatbots for customer support, providing instant responses to queries and guiding users through the customer journey.¹²³ In short, integrating LLMs in marketing enhances efficiency, personalization, and data-driven decision-making, making them valuable tools in the dynamic and competitive marketing landscape.

Human resources

LLMs are being increasingly applied in various human resources (HR) aspects, helping to streamline processes, improve communication, and enhance decision-making and, similarly, to help optimize job descriptions to attract a diverse pool of candidates and ensure clarity in expectations.¹²⁴ They support automating the interview scheduling, saving time, and improving the candidate experience. Specifically, helping to answer common queries from new employees during the on-boarding process, providing information about company policies, benefits, and procedures. They finely contribute to analytics by analyzing employee data, generating reports, and providing insights into workforce trends. They also classify the hiring data to identify areas for improvement in diversity and recommend strategies to enhance diversity in the workforce.¹²⁵ Incorporating LLMs in HR enhances efficiency, personalization, and data-driven decision-making, making them valuable tools for HR professionals in managing various aspects of the employee life cycle.¹²⁶

E-commerce

LLMs play a significant role in the e-commerce sector, contributing to various aspects of online retail. It can generate compelling product descriptions, improving the quality and consistency of e-commerce content.¹²⁷ LLMs analyze customer reviews to provide insights into product feedback content based on user preferences and behavior, enhancing the shopping experience and sentiment.⁵⁴ Subsequently, it enables personalized shopping experiences by tailoring website content, offers, and promotions to individual user profiles, predictive analytics models, forecasting future sales trends, and customer behavior based on historical data. The integration of LLMs in e-commerce contributes to enhanced customer experiences, improved content creation, and management efficiency, and more data-driven decision-making for businesses operating in the online retail space.¹²⁸

Research and academia

LLMs have significantly impacted research and academia, transforming various aspects of the scholarly landscape. LLMs have performed a satisfactory role in automated literature reviews by summarizing and extracting relevant information from a vast corpus of academic papers.¹²⁹ LLMs contribute to the automatic generation of concise and accurate abstracts for research papers, aiding in understanding complex topics, and writing sections of research papers, providing language suggestions, and aiding in the overall structure of academic writing.¹³⁰ LLMs are employed in plagiarism detection tools to identify and highlight potential instances of plagiarism in academic writing. Currently, LLMs are used for automated grading of assignments and exams, providing quick and consistent feedback to students. Besides these, LLMs contribute to building and maintaining institutional knowledge bases, making information easily accessible to researchers and academics.¹³¹ Applying LLMs in research and academia accelerates processes, improves writing and communication, and enhances overall efficiency in various scholarly activities.

Multimodal applications of LLMs or MLLMs

MLLMs can be trained with video, audio, image, and text (Figure 5). Researchers and entrepreneurs have noted the diverse applications of MLLMs and their immense possibilities. MLLMs are increasingly combined with computer vision models for tasks involving text and images, such as images, videos, or audio, to provide a richer and more comprehensive user experience and visual question-answering systems.¹³² The versatility of MLLMs allows them to be applied to many tasks across diverse industries, demonstrating their potential to improve efficiency and provide intelligent solutions. More specifically, LLMs assist in summarizing content within images, providing brief textual descriptions for visually impaired users or those who prefer text-based information.¹³³ They contribute to understanding and interpreting visual cues within chatbot conversations, providing more context-aware responses. For assistance, in integrating textual and visual content for presentations, ensuring coherence and relevance between the spoken or written text and visual aids, MLLMs enhance search engines' capabilities for more accurate results.¹³⁴ MLLMs contribute to creating immersive experiences in virtual or augmented reality by providing natural language understanding alongside visual and auditory elements. Such an essential combination of LLMs in multimodal applications enhances the ability to process, understand, and generate content that spans multiple modalities, providing users with more immersive and contextually rich experiences.¹³⁵ Recently, researchers have developed several MLLM models. One such interesting MLLM is Next-GPT, which can be used as an any-to-any MLLM model.¹³⁶ Another MLLM model is Bliva, which is a very simplistic MLLM. It can handle text-rich visual questions.¹³⁷ Han et al. developed an MLLM-based Chartllama for charts.¹³⁸ Several other MLLMs have been developed for different applications, such as mplug-owl,¹³⁹ Palm-e,¹⁴⁰ Mm-llms,¹⁴¹ M3exam,¹⁴² Kosmos-g,¹⁴³ etc.

Figure 5.

The MLLM’s working principle.

MLLMs in biological macromolecules to biological sciences

MLLMs have been applied in biological science to explore different possibilities. Significant models have been developed to explore the next-generation possibilities. Researchers have developed a GIT-Mol, an MLLM model to explore the possibilities of complex molecular science that integrates text information, images, and graphs. It can predict chemical reactions and compound name recognition.¹⁴⁴ Recently, Lin et al. developed a multimodal deep learning model for multiclass glaucoma surgery and its outcome prediction. The multimodal neural network improves clinical decision-making for postoperative management.¹⁴⁵ Xu et al. developed Protst, a next-generation MLLM model, to explore biomedical texts and protein sequences.¹⁴⁶

Similarly, an MLLM model called MuSe-GNN han developed, which performs gene presentation from multimodal biological graph data.¹⁴⁷ Huang et al. illustrated the application and prospects of an MLLM in dentistry. The researchers explain how MLLMs can shape the future landscape of dentistry.²¹ However, there are many possibilities for the MLLM model to solve complex problems in biological science.

Limitations of LLMs

LLMs, such as GPT models (GPT-3.5 and GPT-4), have achieved remarkable success in various NLP tasks. However, they also come with certain significant limitations (Table 3). LLMs can generate coherent and contextually relevant text, but they often lack deep understanding of the world and common sense reasoning. They can generate nonsensical or incorrect responses in specific contexts.¹⁴⁸ LLMs can inadvertently preserve biases present in the training data, which can result in biased or unfair outputs, especially when dealing with sensitive topics such as gender, race, or religion. Mitigating biases in LLMs remains a significant challenge.

Table 3.

Different limitations of LLMs with their mitigating strategies

Sl. no.	Types of LLM limitations	Description/remarks	Mitigating strategies
1.	Ethical concerns	The responses may be risky, biased, or offensive nature. Having threat of privacy and other security breaches. No established accountability was present for the consequences of model outputs. No consensus on what roles AI should and should not play in medicine.	Refinement to decrease the incidence of undesirable outputs. Formation of governance systems and managing the experts. Fixing of a reporting system for end users to flag the dangerous responses. Consensus-building creativities connecting the patients and medical practitioners.
2.	Coherence	Model outputs are centered on learned associations between the words in spite of considerate input information (queries) used in outputs. Falsified facts are offered as if they were true.	The regenerating model architecture and training tactics used to develop true semantic knowledge. Fine-tuning to exclude presentation of wrong information.
3.	Accuracy	GPT-3 is restricted to data of 570 GB. Models are limited to learning probabilistic associations between words, these are not trained to understand. Training data are obtained from unverified and invalidated resources, websites, books, etc.	Training data validation for insecurity indicators. Fine-tuning used to enhance medical accuracy. Self-improvement over intelligent prompts (like, chain-of-thought).
4.	Recency	The training datasets of GPT do not comprise content created after September, 2021. All concerning datasets necessarily “cut off” at a random time or date.	Assembly of the training data from more current sources. The real-time internet access (e.g., Bing AI, BlenderBot 3, Sparrow).
5.	Transparency and interpretability	It is not clear that how models generate answers from input queries, architectural data, and algorithms (Black Box problem). It is uncertain which parts of the training dataset are gearing, in created responses.	Prerequisite for outputs to cite which portions of the dataset added to the model’s answers section. It has explicable AI research and development.

Moreover, while LLMs excel in understanding and generating text based on context, they may struggle with long-term dependencies or maintaining coherence over extended passages.⁶ This can lead to inconsistencies or inaccuracies in generated text, especially in complex or significant scenarios. The LLMs typically require vast amounts of data for pre-training, which can be expensive and resource-intensive. Furthermore, they may stumble with generalizing to out-of-domain or low-resource domains where training data are limited. While fine-tuning LLMs on specific tasks can improve performance, it often requires careful selection of hyperparameters, task-specific data, and fine-tuning strategies.¹⁴⁹

In addition, fine-tuning may only sometimes lead to optimal performance, especially for tasks with unique requirements or constraints. In the case of safety and ethical concerns, the LLMs have the potential to generate harmful or malicious content, including misinformation, hate speech, or inappropriate material.¹⁵⁰ Ensuring LLMs' safe and ethical use poses significant challenges for researchers and practitioners. Training and deploying LLMs can be computationally expensive and resource-intensive, requiring powerful hardware and substantial infrastructure. It can limit accessibility to LLMs for researchers and organizations with limited resources.¹⁵¹ Addressing these limitations requires ongoing research and development efforts in bias mitigation, robustness testing, model interpretability, and ethical AI frameworks. In addition, interdisciplinary collaboration involving experts from diverse fields, such as linguistics, psychology, and ethics, is essential to foster the responsible development and deployment of LLMs.

Challenges and future prospects

Tokenization in LLMs, although an essential aspect, has its share of challenges. One notable obstacle is the management of out-of-vocabulary tokens. LLMs face difficulties when they encounter words or phrases not present in their training data, affecting their ability to comprehend and generate relevant output. To mitigate this challenge, continuous refinement and adaptation of the model is required to keep up with the ever-evolving nature of language. Furthermore, tokenization may encounter difficulties in capturing nuanced semantic meanings. Ambiguities and polysemy in language can lead to multiple interpretations of a single token, posing challenges for models to discern the intended meanings accurately. Addressing such ambiguities necessitates advancements in contextual understanding and disambiguation techniques. In addition, tokenization can be resource-intensive, particularly in models with extensive vocabularies. Processing vast amounts of data for tokenization may result in increased computational demands, thereby limiting the scalability of models for real-time applications or resource-constrained environments.

Despite the challenges mentioned above, the future of LLM tokenization holds promising prospects. The current research and development activities focus on mitigating the existing constraints and unlock new capabilities. One avenue of exploration is the enhancement of handling out-of-vocabulary tokens. Future LLMs may incorporate more effective mechanisms to adapt to novel language elements, thus reducing the impact of encountering unfamiliar tokens and improving overall language coverage. Another exciting frontier involves advancements in contextual understanding. Research into more sophisticated attention mechanisms and context aggregation techniques can lead to models better equipped to capture subtle nuances in language, enabling more accurate and context-aware tokenization. Improving the interpretability and explainability of tokenization processes is an ongoing focal point. As LLMs become increasingly integrated into various applications, understanding the rationale behind token-level decisions becomes crucial for building trust and ensuring ethical use. Efforts to optimize computational efficiency are also underway. Future LLMs are expected to leverage innovative architectures or strategies to streamline tokenization processes, making them more accessible for various applications and devices. Therefore, while challenges certainly exist, the continuous evolution of LLM tokenization holds immense potential. LLMs can achieve greater accuracy, adaptability, and applicability in diverse linguistic contexts by addressing current limitations and embracing future possibilities. As the field of NLP advances, tokenization within LLMs is poised to play a pivotal role in shaping the next generation of intelligent language understanding systems.

Conclusion

LLMs have revolutionized biological sciences and medicine, resulting in transformative and fundamental applications and faster progress in medicine and different areas of biological sciences. LLMs are helping to generate new hypotheses in these areas. The LLM model also helps clinical decision-making and understanding of possible future outcomes. MLLMs make it faster and provide a broader range of opportunities. There are ample opportunities to research those areas using LLMs or MLLMs. Researchers have explained that the range of possibilities is vast.

However, possible risks generate substantial concerns among researchers, experts, and users. Successful validation of the LLM and MLLM technologies will benefit human society at large. At the same time, ethics, safety, and potential human replacement are the most significant concerns. However, we are hopeful that future researchers will use the technologies in a way that will do justice to society by properly utilizing them.