Artificial intelligence at the pen’s edge: Exploring the ethical quagmires in using artificial intelligence models like ChatGPT for assisted writing in biomedical research

Abstract

Chat generative pretrained transformer (ChatGPT) is a conversational language model powered by artificial intelligence (AI). It is a sophisticated language model that employs deep learning methods to generate human-like text outputs to inputs in the natural language. This narrative review aims to shed light on ethical concerns about using AI models like ChatGPT in writing assistance in the health care and medical domains. Currently, all the AI models like ChatGPT are in the infancy stage; there is a risk of inaccuracy of the generated content, lack of contextual understanding, dynamic knowledge gaps, limited discernment, lack of responsibility and accountability, issues of privacy, data security, transparency, and bias, lack of nuance, and originality. Other issues such as authorship, unintentional plagiarism, falsified and fabricated content, and the threat of being red-flagged as AI-generated content highlight the need for regulatory compliance, transparency, and disclosure. If the legitimate issues are proactively considered and addressed, the potential applications of AI models as writing assistance could be rewarding.

INTRODUCTION

Artificial intelligence (AI) has shown a significant improvement in recent years and is currently imprinting its name in the realm of biomedical research.[1] One fascinating use of AI is in writing activities, where it creates a human-like written text which can be used as content in research papers.[2]

The usage of AI models such as chat generative pretrained transformer (ChatGPT), which have made remarkable advancements in recent years, for various tasks, including research and writing assistance, is increasingly widespread. These AI models use natural language processing (NLP) and advanced algorithms, which give these models advanced powers to understand the textual prompts and inputs in the same way as their counterparts, human beings can. These NLP-powered AI tools assist in translating technical jargon into understandable plain language for the audience. These prompts and inputs serve as initial commands or instructions given to the AI model, which helps them to provide a guided output that aligns with the context of inquiry and sounds logically correct.[3,4] Thus, making it an ideal candidate for various applications, including virtual and writing assistance.[5]

Incorporating AI in most of the recent writing assistance software gives immense possibilities for increasing efficiency and productivity in biomedical research. However, its use as writing assistance in biomedical research highlights some critical ethical issues. As researchers, we may get fascinated with its power to create excellent writeups, which may sound logically correct but also raise suspicion about the reliability or accuracy of AI-generated content. The researcher must ensure that the information retrieved from such AI models aligns with the complex findings of biomedical research if they are to be used in writing a study. It is also desired that the output needs to be more informed and accurate.[6]

This narrative review aims to shed light on ethical concerns about using AI models like ChatGPT in writing assistance in the healthcare and medical domains.

WHAT IS CHAT GENERATIVE PRETRAINED TRANSFORMER?

ChatGPT, is an advanced language model developed by Open AI.[7] The built of ChatGPT is designed based on GPT, i.e., GPT architecture, which intends to produce text responses that resemble those of a human being having a conversation.[8]

As the name “ChatGPT” suggests, it is pretrained with an enormous amount of textual data from the internet. This pretraining helps the AI model to learn the semantics, structure, and language patterns of human beings. This, in turn, can help yield a logical, guided response that adheres to the context of inquiry.[9]

ChatGPT, like AI models, are experts in understanding human queries and generating natural language responses mimicking human conversation. This allows them to provide logical answers in an interactive dialogue based on information provided as prompts and inputs, making it suitable for applications such as chatbots and virtual or writing assistance tools.[10]

ROLE OF PROMPTS AND INPUTS IN ARTIFICIAL INTELLIGENCE-ASSISTED BIOMEDICAL RESEARCH WRITING

In ChatGPT, prompts and inputs are the initial information or instructions provided to the model to generate a response. They serve as the starting point for the model to understand the desired context and generate text accordingly.[11]

Prompts

Prompts are certain cues or instructions which, when given to ChatGPT, aid in generating a guided response. A prompt can be input such as text, question, or even a statement, giving AI the direction for processing and yielding a desired output. For example, a prompt could be “Can you please search pathophysiology of COVID-19?” or “Can you please search the role of genetics in COVID-19 susceptibility.” These prompts serve as an initial guide for the AI model to understand the context of inquiry and give a relevant output. Hence, prompts have the following role in AI-assisted writing, i.e., act as a guide to provide the necessary context and specific instructions for the AI model, act as a direction for the AI model, and serve as a means of quality control. Finally, it acts as a mechanism for ethical control and oversight.[4]

Inputs

Similar to prompts, inputs are additional details or instructions communicated to ChatGPT during a chat or encounter. These comprise specific phrases from the earlier instructions, sentences, or messages and any pertinent details required to direct the model if the intended outcome deviates from the input. Inputs can also be employed when output is necessary for extra context. The primary distinction between prompts and inputs is that the latter advances the current dialogue and aids the model in comprehending it. Here’s an illustration: Assume that the pathophysiology of COVID-19 has been discussed in the ongoing conversation with AI. In that situation, the inputs would be user cues like additional inquiries about specific genes or further questions related to the topic.[12]

Effective use of prompts and inputs by the users can help generate guided ChatGPT’s responses and also make sure that the AI understands the desired context and provide relevant output. These shape the conversation between the user and the AI model to generate logical and appropriate context-related responses.[3] It is vital to craft prompts and inputs carefully to ensure clear communication and achieve the desired outcome from the model.

ETHICAL CONCERNS THAT MUST BE PROACTIVELY ADDRESSED WHILE USING ARTIFICIAL INTELLIGENCE MODELS AS WRITING ASSISTANCE

Data limitations

AI models are trained on large datasets, and the quality and representativeness of the training data can impact their performance. In biomedical research, the availability of comprehensive and high-quality datasets can be limited, leading to potential biases or incomplete understanding of certain aspects.[10,13]

Lack of contextual understanding

AI models like ChatGPT generate responses based on patterns learned from vast text data. However, they may need a deeper understanding of the context, nuances, and intricacies specific to biomedical research. In some cases, they can generate responses that can be plausible sounding but inaccurate or misleading information. Even in some cases, there can be social Bias and hallucinations.[14,15] This underscores the importance of human verification and critical evaluation to ensure the validity and reliability of the research findings.[16]

According to American Medical Writers Association, AI is a man-made innovation that is susceptible to many limitations and potential errors, and the data used to train these models represents the accuracy and reliability of its response. For instance, if AI tools are only familiar with information about superficial spreading melanoma and not with other types of melanomas, the content they produce may be incomprehensible or, to put it another way, irrelevant to generalized the finding for other melanomas because superficial spreading melanoma only accounts for about 70% of all melanomas.[17]

Dynamic knowledge gaps

Biomedical research is a dynamic field with new discoveries, advancements, and updates emerging regularly. AI models may not have access to the most up-to-date information as they are trained on an existing large dataset; also, they lack the ability to access external data or the ability to browse the internet, leading to outdated or incomplete responses.[10] Staying abreast of the latest scientific developments is crucial, and human researchers play a vital role in verifying and updating the information. Relying solely on AI-generated content without human verification can lead to potential inaccuracies, misinformation, or flawed conclusions in research.[18]

Limited discernment

Biomedical research often requires critical analysis, interpretation of complex data, and making informed judgments. Currently, all the AI models are in the infancy stage and lack the ability to reason, evaluate evidence, or apply expert review in the same way human researchers can. This can generate responses that may not align with the field’s best practices or accepted standards.[19]

Lack of responsibility and accountability aptitude

In contrast to humans, AI lacks consciousness and the capacity to accept responsibility for its actions. In other words, AI models like ChatGPT can not be held accountable or responsible for malicious information if generated. Therefore, all the information or content generated by AI should be meticulously reviewed, fact-checked, and validated by human researchers or subject experts before including in their research work. Meticulous human review of the generated content is crucial for the reliability and accuracy of the research findings and for preventing the dissemination of potentially faulty or flawed results.[20]

Privacy and data security

The use of an AI model as an aid in writing research papers may be alluring to young researchers to increase their productivity. However, it should be kept in mind that it involves providing information in the form of prompts and inputs to retrieve the desired outputs; this information can sometimes be confidential and sensitive and may be about research participant data, patient data, medical reports, or proprietary research data from clinical trials, and electronic health records. It is clearly stated in most AI-modeled websites that the user data may be stored for training and quality purposes. The researcher must adhere to the laws of data use and make sure that the use of such data is as per the informed consent of the participants, privacy policy, and ethical guidelines. Obtaining appropriate permissions, maintaining data confidentiality, and protecting patient privacy are paramount.[21]

According to the European Union (EU), even though The EU General Data Protection Regulation (GDPR) partly regulates these AI models and has a promising future in automated decision-making in healthcare services and medical research, many ethical and regulatory issues remain to be resolved. The GDPR protects data subjects and processes personal data against entirely automated decision-making.[22]

In India, NiTi Aayog presented the first official AI document in 2018, named “National Strategy for AI.” This strategy document was based on the idea that, given its strengths and features, India has the potential to become one of the leaders on the global AI map with a distinctive brand of “#AI for All.” Through this document, the NiTi Aayog focused on five sectors envisioned to benefit the most from AI: Healthcare, Agriculture, Education, Smart Cities and Infrastructure, and Smart Mobility and Transportation. It was emphasized that as data are one of the primary sources of inspiration for AI solutions, it is essential to handle the data appropriately and uphold privacy and security.[23]

The 2023 Indian Council of Medical Research (ICMR), through its Department of Health Research (DHR)-ICMR AI Cell, has developed “Ethical Guidelines for Application of AI in Biomedical Research and Healthcare,” which primarily addresses ethical principles for AI technology in healthcare, including autonomy, safety and risk minimization, trustworthiness, data privacy, accountability, and liability, optimization of data quality, accessibility, equity, and inclusiveness. In addition, these guidelines offer guiding Principles for Stakeholders engaged in developing, validating, and deploying AI systems and their ethical review processes in medical AI.[24]

Despite the NiTi Aayog Strategy paper and ICMR AI guidelines. India is still waiting for a regulatory framework for AI-enabled smart tech platforms, like ChatGPT, which would offer a detailed framework for bias in algorithms and copyright. Therefore, researchers are encouraged to use these models while taking the required security measures to protect the privacy and confidentiality of such data.[25]

Lack of transparency in artificial intelligence development and bias

Developing an AI model is a complex and highly specialized task, which involves collecting and preprocessing the data to be trained to the model and, at last, building and training the algorithm. The diversity and representativeness of data used to train these models are paramount. In addition, the data should also be free from any bias. Similarly, when AI models such as ChatGPT are considered for biomedical research writing, the model is expected to be trained with data from various demographics, populations, medical conditions, and a broad range of samples. If the data used for training is not diversified or is biased toward a specific group or circumstances, a range of biases may arise. These models might be unable to delineate such instances due to their faulty training and may not generalize well to various populations or circumstances.[26] Good research should be mindful and free from potential limitations or biases. Improperly trained AI models could impact the fairness or accuracy of their responses and decision-making.[27]

Lack of nuance, style, or originality

Although ChatGPT 3 and 4 have improved learning opportunities, they still have several drawbacks. Based on user-input commands and prompts, ChatGPT creates a guided response. Given that they are trained primarily from text data that has already been collected and are not explicitly programmed with concepts of creativity or style, these AI models may need help to produce original content. The model can learn from the training data and provide guided output by combining data from many sources and creatively rephrasing it. Consequently, it might result in something other than cutting-edge or creative knowledge.[28]

Authorship issues and unintentional plagiarism

The growth of biomedical research depends on current information. The researchers must comprehend how important it is to acknowledge those who contributed to the study or from whom the central concept was drawn. Comparable prompts and inputs may result in identical outputs when AI assists writers. Unintentional plagiarism may result when these outputs are sent to publications without being carefully reviewed, supervised, or assessed.[29] Similarly, one incident where the ChatGPT was included as a co-author of the article was recorded. This practice is disputed since many scientists oppose it, and many science publications forbid the ChatGPT from being added as a co-author on research articles.[30] To assure appropriate authorship and prevent potential problems with plagiarism or intellectual property breaches, researchers must clearly define the responsibilities of human researchers and AI systems in the study process.

As stated in the World Association of Medical Editors (WAMEs) Recommendations on Chatbots and Generative AI in Relation to Scholarly Publications, chatbots cannot be authors. Authors are also advised to use chatbots transparently, disclose their use when doing so, and take responsibility for material provided by a chatbot in their paper. In addition, WAME emphasized that editors should have access to tools for spotting material produced by AI.[31]

Also, as per International Committee of Medical Journal Editors (ICMJE) recommendation, “the use of AI-assisted tools in the creation of submitted work should be disclosed by authors.” In addition, they also state that these chatbots, i.e., ChatGPT, cannot be listed as authors because they cannot be held responsible for the accuracy or integrity of the content generated and also cannot be relied on for the originality of the submitted work; thus author should be solely responsible for the content included.[32]

Taylor and Francis, in their clarification regarding “The Responsible Use of AI Tools in Academic Content Creation,” states that these tools cannot be listed as an author but can be acknowledged for their contributions.[33] Similarly, according to Springer, AI programs should be discussed in the methodology portion of a research paper but cannot be authors.[34]

The threat of falsified and fabricated content

As pointed out in the review, a vast amount of text data from many sources, including the internet, is fed into AI models like ChatGPT to enable them to provide a guided answer based on their learned algorithms. These data need to be more accurate and trustworthy and contain biased and inaccurate information. Although these models produce text or other output in response to the user’s inputs and cues to assist them in providing meaningful information, it must be remembered that they are devoid of innate knowledge and human-like awareness of the outside world. It depends on the data pattern on which it was trained, which may only sometimes be factually correct. Consequently, fake or illogical content can be generated due to hallucinations without adequate oversight and fact-checking procedures.[35,36]

These hallucinations are the primary setbacks in using AI models like ChatGPT in literature searches. Due to time-consuming and complex processes, ChatGPT is becoming an appealing tool for doing systematic literature reviews, where the researcher instructs these models to conduct a literature search on his or her behalf based on inputs reflecting the research question. In response, ChatGPT provides the necessary output in logical and persuasive language, luring researchers to include these findings in their literature studies.

There is mounting evidence that many of the contents generated by these literature searches are fictitious or the product of ChatGPT’s hallucinations.[15,37] Utilizing such erroneous sources will affect the findings of future systematic literature reviews and biomedical literature and destroy the essence of evidence-based decision-making.

Regulatory compliance

Various ethical principles and regulations apply to all biomedical research and surveys. The Institutional Review Board carefully examines them to ensure that the research’s integrity and the rights of human subjects are upheld. To protect the rights of participants and themselves, researchers must consider guidelines and ethical principles when utilizing AI in writing a study.[38]

Transparency and disclosure

Researchers must be upfront in disclosing the use of AI models in the research process, which is crucial when AL models are used as writing aid. This entails admitting the use of AI systems in research articles, outlining any restrictions or potential biases related to content produced by AI, and offering comprehensive justifications of how the AI support was applied.[39,40]

The threat of being red-flagged as artificial intelligence-generated content

Now that AI models such as ChatGPT have been developed and used for a while, there is a new threat of content being reported as AI-generated. Extensive use of ChatGPT-like models may result in flooding low-quality articles with questionable journal findings. These low-quality articles incorporate certain biases or even content generated from AI model hallucinations.[41] In addition, many professionally available websites, such as Originality AI, Copyleaks AI, AI Detector Pro, GPT 0, 0 GPT, Turnitin, iThenticate, and Content at Scale, use the same algorithm to detect AI-generated content to a reasonable extent.[42] As per “The Guardian,” Scott Aaronson says that to prevent cheating Open AI is currently working on watermarking the outputs.[43,44] Incorporating such outcomes without careful assessment may lead to false conclusions about AI-generated content warnings by various tools. Thus, researchers should remember that AI-generated content cannot be dependable or accurate.

The dilemma of artificial intelligence in manuscript peer review

The usage of AI models like ChatGPT for manuscript writing and peer review is currently the subject of heated discussion. Even though peer reviewing takes time, many researchers view it as an honor because it gives them a chance to critically assess a manuscript, improve the final manuscript, and reject the unworthy while allowing the reviewer to learn more about the research process.[45]

On the other hand, AI can peer evaluate an article in a matter of seconds, negating the necessity for a time-consuming human peer review. Peer review, however, is more than merely an approach for discovering errors or altering language; it’s also crucial for making the best decisions possible by evaluating the topic in light of its contemporary implications and improving how it’s presented to the community. AI models don’t have the subtleties of a human-based peer review process, in-depth knowledge, or decision-making abilities. This might lead to sloppy judgment and the loss of numerous manuscripts that could otherwise be valuable for their profession. Hence, careful consideration should be made while reviewing the manuscript with the help of AI models. However, they reduce the time taken but cannot be the sole means for judging a manuscript’s worth. As a result, although using AL models to assess a paper takes less time, doing so should not be the only way to determine whether or not a manuscript is valuable.[45]

Real-life hurdles in incorporating chat generative pretrained transformer for research and writing purpose

An in-depth review of the currently published papers reveals various difficulties with using ChatGPT for writing and research [Table 1].

Table 1.

Case scenarios and studies using ai models for research and writing purposes and problems encountered by authors

Author, year	Problems identified
Alkaissi and McFarlane, 2023[15]	Only 8 of the 50 retrieved DOIs existed, while the ChatGPT provided bogus papers in 66% of cases
Haman and Školník, 2023[37]	When pressed to explain the findings, ChatGPT presented outdated references with fraudulent PMIDs that did not exist in real life
Laudicella et al., 2023[46]	When asked about the role of ChatGPT in nuclear medicine and radiology? The model stated that it has no direct involvement in nuclear medicine or radiology and is designed to generate human-like language based on given cues
Jansz et al., 2023[47]	ChatGPT did an outstanding job of summarizing the subject at hand. There were no citations to relevant literature. ChatGPT incorrectly described periorbital edema in SLE as a “common” symptom, even though uncommon
Schussler et al., 2023[48]	Entire sections of the responses could be utterly incorrect, even though they appear plausible
Schuppe et al., 2023[49]	ChatGPT performed admirably in editing and revising the case report. The model, however, could not generate references and citations for the text it generated. The model also generated bogus references with plausible authors, titles, and journal names
Raxwal et al., 2023[50]	In terms of original arguments or concepts, it falls short. Furthermore, ChatGPT cannot interpret context or connect unrelated pieces of information
Hegde et al., 2023[51]	The authors should double-check the content provided by ChatGPT since it has the potential to deceive them with well-written writing and false facts
Lantz, 2023[52]	Upon manual search reality check, the majority of the references were obsolete, and four of the seven references provided by ChatGPT were fictional. The authors stressed that ignoring these evident citations and reference issues is impossible
Nachshon et al., 2023[53]	Even in a simple case scenario, ChatGPT cannot assist in creating accurate scientific reports, despite providing some essential background knowledge on the area of interest
Akhter and Cooper, 2023[54]	ChatGPT provided a good general introduction and summary of the topic of interest. However, it was insufficient to generate the discussion section because it couldn’t find relevant literature and had a data training limit of 2021
Mago and Sharma, 2023[55]	ChatGPT-3 accurately defines the disease, radiographic characteristics, and anatomical landmarks. The content of the oral and maxillofacial pathologies was limited to major characteristic radiographic features. One of the two abbreviations was incorrectly answered. ChatGPT-3 is less detail-oriented, making the data susceptible to infodemics and medical errors

DOIs=Digital Object Identifiers, PMIDs=PubMed IDs, SLE=Systemic lupus erythematosus, ChatGPT=Chat generative pretrained transformer

Highlighting these findings may assist researchers in becoming aware of ethical considerations that must be addressed while using these cutting-edge AI systems.

Finally, the authors of this review advocate for a balanced approach to manual writing and AI aid. Although utilizing ChatGPT to produce scientific publications quickly and effectively is a possible option, there is some concern that it might discourage genuine researchers from using their manual writing skills. Furthermore, using AI models to generate text, such as ChatGPT, may foster a sense of dependency on these advances, weakening the critical thinking abilities that come with handwriting over time. A balance must be achieved to keep manual writing abilities growing while employing AI.

CONCLUSION

The integration of AI writing assistance in biomedical research has the potential to enhance productivity and efficiency. However, addressing the ethical concerns associated with its use is imperative. In addition, it is impossible to forbid their use outright or intentionally choose not to utilize them because they can significantly enhance many aspects of biomedical research. Critically reviewing the information obtained for its accuracy, reliability, and transparency can ensure the responsible and ethical utilization of AI models like ChatGPT.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.