Welcome, AI overlords?

Despite its pervasive presence today, it’s remarkable in retrospect that ChatGPT, the chatbot from the firm Open AI available to the public as a web app, dates back only to November 30, 2022. Two months after its launch, ChatGPT reached 100 million active users, making it at the time the “fastest-growing consumer application in history” (1). Among the early users were members of the scientific community. By January 18, 2023, at least four scientific papers had been published with ChatGPT as an author (2). One of the four, titled “Rapamycin in the context of Pascal’s Wager: generative pre-trained transformer perspective” (3), was published to provide a precedent for recognizing the potential for large language models such as ChatGPT to make sufficiently “meaningful contributions” to academic work to justify authorship. Ironically, as reported in the Author Contributions section of the paper, when ChatGPT was asked by Alex Zhavoronkov whether it should be included as a coauthor, it responded with multiple compelling reasons as to why it should not.

May R. Berenbaum.

By February 2023, along with many other high-profile scientific journals, the PNAS journals announced their policies for ChatGPT and generative AI (https://www.pnas.org/post/update/pnas-policy-for-chatgpt-generative-ai). These policies specified that “The software cannot be listed as an author because it does not meet the criteria for authorship and cannot share responsibility for the paper or be held accountable for the integrity of the data reported.” Moreover, authors using AI software to prepare a manuscript must describe details about its use in the manuscript and, as well, must recognize they are “solely accountable for, and must thoroughly fact-check, outputs created with the help of generative AI software” (https://www.pnas.org/author-center/editorial-and-journal-policies#authorship-and-contributions).

Restrictions on the use of ChatGPT became a necessity at least in part because of the limitations of the technology. Because it was trained on information available only through 2021, ChatGPT-4 does not access the vast amounts of information generated more recently (4). AI language models also can on occasion plagiarize by paraphrasing too closely text on which it was trained. As well, as a consequence of the intrinsic design decisions of large language models, ChatGPT is prone to “hallucinate,” i.e., fill in content gaps with plausible but not necessarily accurate information, a profound shortcoming in the context of scientific publishing. Moreover, it is also limited compared with conventional human authors in that it can’t provide personal insights based on experience, nor can it create a decent joke (5), although this shortcoming is not likely viewed as such in the context of scientific publishing. And I’m sure that people will be thrilled to learn that, despite initial doubts (e.g., ref. 5), ChatGPT-4 seems to have mastered recognizing and imitating sarcastic tones.

While these quirks may reduce the utility of ChatGPT for generating manuscript text, other attributes are more concerning. The ability of AI to perpetuate misinformation has led some to believe that, with time, AI may “pose an existential risk to humanity” (e.g., ref. 6)—possibly an even greater risk than “climate change, pandemics, asteroid strikes, supervolcanoes, and nuclear war combined” (7). Training AI with contemporary texts riddled with gender bias, political extremism, and conspiracy theories could through recursive self-improvement create a feedback loop that perpetuates past bad practices well into a dismal future. It’s probably little consolation for those worried about AI that, when asked, ChatGPT denies having any intention of “taking over the digital world,” explaining that “As an AI model developed by OpenAI, I don’t have any intention or capability of taking over the digital world…. I am designed to work within the constraints set by my creators and do not have the ability to operate independently or make decisions that would go against their guidelines” (8).

Concerns about the end of humanity are not entirely without merit, but they’re also not entirely new; across the centuries, many life-changing technological innovations without which modern life would be unthinkable did in their early days engender fears about the erosion of the quality of life, if not the end of human life as we know it. Historically, the key to averting the end of humanity has been, across generations and technologies, to create guidelines to mitigate the risks. In the case of AI, this mitigation involves ensuring that AI is trained to align with human values.

If its powers can be harnessed, ChatGPT has tremendous potential to do what technological innovations have often been designed to do—to make life easier for people. In the scientific publishing enterprise, there’s considerable room for making life easier. PNAS is interested in finding out what authors consider to be the most time-consuming or frustrating aspects of scientific publishing and exploring whether new technology can reduce this burden. ChatGPT’s strength in summarizing and analyzing texts, for example, is suggestive of its potential utility in writing abstracts and selecting key words. Interpreting and summarizing data, checking references, proofreading for spelling and grammar, and formatting according to journal specifications are all potential time-saving tasks that can be carried out with ChatGPT. As well, AI has the potential to democratize the preparation of manuscripts by removing or reducing language barriers. The overwhelming domination of the scientific literature by the English language disfavors that portion of the scientific community that lacks fluency. The capacity of ChatGPT to translate texts into multiple languages, although imperfect, could go a long way toward lowering language barriers for nonspeakers of English and promote their participation in high-impact scientific publishing. All these positive outcomes, though, depend in large part on figuring out a way to reduce GPT’s predilection for making up “facts” in the absence of actual relevant information.

One important caveat on LLM technology in writing scientific text is that, in these early stages, its use in manuscript preparation really should be disclosed for the benefit of editors, reviewers, and readers. There have been inadvertent disclosures, as, for example, when authors fail to notice stray phrases left behind in their text by ChatGPT, which subsequently may escape notice by reviewers, copyeditors, or proofreaders and end up mystifying readers in published papers. One such phrase, “regenerate response,” which is the name of a button to press to direct ChatGPT to open a new page and restart, has been found by University of Toulouse computer scientist Guillaume Cabanac in more than 30 published papers (9). Describing how ChatGPT is used in conducting a study or preparing a manuscript increases transparency, just as describing how other tools are used (e.g. statistical packages for data analysis). Full disclosure of the use of AI tools also can help the scientific community learn to recognize and understand their strengths and limitations. This perspective underlies recent modifications of PNAS policy on the use of artificial intelligence to include the statement, “Use of AI software, such as ChatGPT, for manuscript preparation, including drafting or editing text, must be noted in the Materials and Methods section (or Acknowledgments, if no Materials and Methods section is available) of the manuscript.”

PNAS is far from the only journal adjusting to generative AI as a new force in publishing, In early October, a group of editors of bioethics and humanities journals published a “preliminary set of recommendations about generative AI in scholarly publishing” (10). These recommendations converge with recommendations being developed and adopted by science journals and include exclusion of generative AI tools or LLMs from authorship status and transparent disclosure by authors about the use of these tools. They also call for the development of methods for identifying AI-generated text if authors fail to document its use and they conclude with a reminder that professional norms best evolve in the context of broad public conversation.

We’re mindful at PNAS that the enabling aspects of technology must comport with cultural values and AI tools must be developed with this constraint in mind. After all, this has been the challenge throughout the evolution of scientific publishing. Although the invention of the movable-type printing press in the 15th century was critical to fostering the Scientific Revolution, even mechanized printing had its detractors. Some authors were reluctant to embrace this new technology out of fear that the newfangled mechanical devices would introduce errors for which they as authors would be responsible, at the price of their reputation (11). ChatGPT doesn’t care about its reputation, but twenty-first century authors should take a page, as it were, from their fifteenth century predecessors and recognize it’s their reputation on the line when they use these newfangled AI tools.