The rise of artificial intelligence: addressing the impact of large language models such as ChatGPT on scientific publications

INTRODUCTION

Artificial intelligence (AI) permeates our modern society. Chatbots are being used by commercial companies as an interface with their customers, and Internet search engines rely heavily on AI to function. Smart homes, navigation systems and immigration clearance using facial recognition are other examples of the use of AI in our daily lives. In general, AI is an essential element for biotechnology. In biotechnology and research, the applications of AI include drug target identification, image screening, predictive modelling of disease states and outcomes, assessment of functional and structural genomics, metabolomics and proteomics[1] among others. In scientific writing, AI is used in literature search in biomedical databases such as PubMed, and it is present in a rudimentary form in word processing software that researchers use to write scientific manuscripts. AI can be broadly classified into narrow AI, general AI and artificial superintelligence (ASI). Narrow AI is created to solve specific problems. General AI aims to create machines capable of reasoning and thinking like humans. ASI would be capable of outperforming humans. Currently available AI systems are considered narrow AI.

LAUNCH OF CHATGPT

We entered uncharted territories in scientific writing with the launch of Chat Generative Pre-trained Transformer (ChatGPT; available from: https://chat.openai.com/chat) on 30 November 2022 by OpenAI, an AI research and deployment company. ChatGPT is a large language model (LLM) that has the ability to generate human-like text in a conversational way, including complete paragraphs or even essays, based on specific prompts and context, by mimicking the statistical patterns of language from a huge database of text collated from the Internet. It can potentially write an entire scientific manuscript. Using ChatGPT creates the illusion of conveniently having an expert readily at hand to answer one's questions at all times. The current version of ChatGPT is a prototype that was trained on dataset that is updated up to 2021. It is not connected to the Internet for real-time updates and does not have access to new knowledge derived after 2021. Therefore, it is to be expected that there will be data gaps in ChatGPT, in terms of new information that becomes available after 2021. In future, the currency of knowledge may be less of an issue if there is continuous input of new data for software training and development, similar to how academics and professionals continuously acquire new knowledge and update themselves. The capabilities of LLMs are limited to the textual training data used, and this can result in problems of reliability and bias. In addition, each LLM has a certain amount of memory, and hence can accept only a finite amount of input to form the context for output generation. There are several issues that need to be properly addressed even at this nascent phase of its development.[2] Concerns have been raised about its role in scientific writing, including whether it can be considered as a co-author, its potential negative impact, such as whether it will result in factual inaccuracies, plagiarism or even outright scientific fraud, and whether it facilitates or blunts the academic development of an individual.

Issue of authorship and transparency

ChatGPT has been used to draft scientific abstracts and papers, and in the process, it has even been listed as a co-author in academic publications.[3,4,5] The International Committee of Medical Journal Editors (ICMJE) recommends that authorship be based on meeting the following four criteria: (a) substantial contributions to the conception or design of the work, or the acquisition, analysis or interpretation of data for the work; (b) drafting the work or revising it critically for important intellectual content; (c) final approval of the version to be published; and (d) agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.[6] Thus, it is clear that even if ChatGPT or more sophisticated AI programs were used to write the entire manuscript in future, they would still fall short of the eligibility criteria, especially with respect to the issue of personal accountability by authors. AI tools cannot assume such a responsibility. There is consensus by many researchers and journals that AI models cannot be credited as authors,[7,8] and this is the position that the Singapore Medical Journal holds. Transparency and integrity are crucial elements in scientific writing. A submitted academic manuscript must be the original work of the authors, and the wholesale use of ChatGPT without formal acknowledgement is akin to plagiarism. If ChatGPT or any other LLM is to be used as part of the manuscript preparation, there needs to be clarity on how it is being used when the manuscript is written, and whether and how data derived from ChatGPT and other LLMs are further verified. This must be appropriately documented either in the methods or the acknowledgement section of the manuscript. Such transparency is important to prevent erosion of trust when the manuscript is being appraised. It has also been advocated that as an additional safeguard, AI output detectors be incorporated into the editorial process to alert journal editors and reviewers — similar in concept to current approaches of using AI algorithms to screen for plagiarism by checking the similarity index — and to check for bot-like behaviour in the Internet.

Tackling the issues of inaccuracies and quality

There are concerns about data gaps, factual inaccuracies and bias when ChatGPT is used by authors to draft manuscripts. Although the text written by ChatGPT may appear credible, it has been shown that the generated information can be pure confabulation, with a combination of both facts and fabricated information, or totally fictitious pseudoscientific material. This has been described as artificial hallucination.[9] In future, continuous or regularly scheduled updates of the AI software may help to attenuate this problem of knowledge gaps in the software. Even if there is regular training and updating of the AI software, there may still be a lag phase. Bias in scientific interpretation and writing may also exist, and this could be related to the training of the AI algorithm. It has been suggested that the way the ChatGPT was trained created a left-wing, liberal bias in its responses.[10] The ease of accessing LLMs may result in erosion of skills and expertise needed for high-quality academic publication and promote proliferation of low-quality scientific papers.

Therefore, concerns about integrity, accuracy and deterioration of quality with the use of ChatGPT and similar programs in academic writing are justified. However, problems such as academic bias, inaccuracies, ghost authorship, plagiarism and fraud may still arise even in the absence of AI tools like ChatGPT. It is thus crucial that the concept of research integrity be continuously emphasised to all researchers, journals screen and sieve out submissions of dubious quality, and academics and reviewers develop and retain the skillset for critical appraisal, discernment of information and source verification. Clinicians and scientists who read scientific publications must be able to think critically and not accept information at face value. We have already pivoted away from eminence-based medicine to evidence-based medicine. Human verification of information generated by AI remains important. In reality, critical thinking is important not only within academia, but also generally in life, and one needs to be able to separate facts from myths, half-truths, disinformation, pure propaganda and lies. Having insights on how an AI program was developed would allow one to have a better appreciation of its limitation and the potential for bias and errors. A case in point is the knowledge that the current version of ChatGPT has not been trained on new information derived after 2021, and researchers would thus know that ChatGPT cannot be a trusted source for information after 2021. However, a full understanding of the developmental process of the AI model, including the type and extent of information used in its training, and the potential personal bias of people involved in programming may not always be possible, with such systems existing as a black box due to proprietary reasons.[11] Human verification of the accuracy of AI outputs is crucial, and AI should be considered an additional resource to obtain data, with potential for gaps or even inaccuracies, and not a purveyor of absolute truth.

CONCEPT OF COLLABORATIVE INTELLIGENCE

ChatGPT or other LLMs can facilitate the conduct of research by compiling pre-existing information. For instance, it can be used to draft a study protocol or format the manuscript after completion of the study, which speeds up the writing process. AI can also help improve the language of non-native English speakers and allow translation of an English manuscript to other languages, thus improving access to non-English readers and broadening the reach of journals. However, AI cannot generate new knowledge out of a vacuum when the research has yet to be conducted. Well-trained researchers with deep domain expertise are the ones at the forefront to conduct novel research, generate new findings and achieve scientific breakthroughs. The human interface remains essential for scientific progress. LLMs can be writing assistants and journal assistants but cannot replace human authors or human editors. At the same time, continued training of LLMs will and should occur, so that they can be further improved to help the writing and publishing process. We need to optimise collaborative intelligence, where humans and AI work together, to ensure transparency, integrity and scientific rigour.

CONCLUSION

AI can be a boon or a bane. While AI is an enabler and a game changer, there exist privacy and ethical concerns, possibility of bias, potential legal liabilities and concerns about validity.[12] In the novel ‘Origin’ written by Dan Brown, a terminally ill Edmond Kirsch instructed Winston, the AI personal assistant he invented, to maximise viewership and hype up an important announcement he was going to make on the Internet. Winston then orchestrated Kirsch's death during his presentation to make him a martyr, thus boosting online viewership from 3 million to more than 200 million people. AI can facilitate progress, and hence, we should continue to embrace AI and establish collaborative intelligence, especially as it continues to evolve, moving from narrow AI to higher levels of function, such as general AI, or even ASI in future. However, it is important to set up guardrails and a framework to guide its application, so that the potential of AI can be harnessed appropriately.