Safeguarding authenticity for mitigating the harms of generative AI: Issues, research agenda, and policies for detection, fact-checking, and ethical AI

Summary

As the influence of transformer-based approaches in general and generative artificial intelligence (AI) in particular continues to expand across various domains, concerns regarding authenticity and explainability are on the rise. Here, we share our perspective on the necessity of implementing effective detection, verification, and explainability mechanisms to counteract the potential harms arising from the proliferation of AI-generated inauthentic content and science. We recognize the transformative potential of generative AI, exemplified by ChatGPT, in the scientific landscape. However, we also emphasize the urgency of addressing associated challenges, particularly in light of the risks posed by disinformation, misinformation, and unreproducible science. This perspective serves as a response to the call for concerted efforts to safeguard the authenticity of information in the age of AI. By prioritizing detection, fact-checking, and explainability policies, we aim to foster a climate of trust, uphold ethical standards, and harness the full potential of AI for the betterment of science and society.

Graphical abstract

Biocomputational method; Bioinformatics; Biological sciences; Computational bioinformatics; Natural sciences; Neural networks; Artificial intelligence; Artificial intelligence applications

Introduction

The advent of generative artificial intelligence (AI) tools has generated a wave of both excitement and apprehension within the scientific community. While the potential benefits are immense, they are often overshadowed by significant challenges. Generative AI tools are renowned for their remarkable ability to produce compelling content. However, the authenticity of such content remains a pressing concern. Among the numerous AI tool providers, ChatGPT¹ and Google Bard² stand out as leading pre-trained transformer-driven tools with formidable generative capabilities (hence the GPT). Yet, at present, their statements lack the backing of credible references. This raises concerns about the factual accuracy of their convincing output. The scientific community bears the responsibility to distinguish generated content from genuine scientific endeavors, particularly when such material carries the risk of being used as false evidence in scientific publications.

Generative AI tools are trained using large language models (LLMs) on a vast array of online resources and other datasets. While it is not asserted that all generated content is erroneous, the extent of factual knowledge versus false claims remains unclear. Further complicating the matter, certain transformer-based tools, such as graph neural networks (GNNs), widely employed in drug discovery, provide drug recommendations without transparency or reproducibility trails. Reproducibility issues stem from instances where authors fail to elucidate their findings, accepting the black box³ default without elaboration. Supporting or endorsing such research during the peer-review process is not considered sound practice, as it plays a critical role in the scientific production process. Accepting the current default behavior of these tools could impede future discoveries.

In essence, the emergence of generative AI tools has introduced a new frontier in scientific research, presenting both extraordinary opportunities and formidable challenges. The scientific community must navigate this delicate balance with prudence, ensuring that the benefits of these tools are harnessed responsibly while maintaining the integrity of scientific inquiry. To proceed, in next section, we present our perspective on three crucial aspects that we believe are mandatory to counter the issues mentioned: (1) the detection of inauthentic content and fake science; (2) computational fact-checking versus human verification; and (3) the transparency and reproducibility of science.

Our perspective on safeguarding authentic science

Inauthentic content detection

The alarming rise of fake science has ignited a call for action.^4,5 The scientific community must collectively embrace computational methods to detect inauthentic content. As the world united to combat the global pandemic (followed by an infodemic), now is the time for the scientific community to accelerate the development of algorithms, methods, approaches, and heuristics capable of identifying generated and fake content. Prior to the advent of generative AI, research on fake news^6,7 and fake science^8,9 had already gained significant traction. However, the emergence of powerful generative tools underscores the need for scientists to intensify their efforts to combat misinformation, disinformation, and fake science that may arise from the misuse of these tools. Such endeavors are already underway. Abburi et al. proposed a novel ensemble approach for detecting AI-generated text.¹⁰ Others have comprehensively surveyed the research landscape and revealed the potential of detecting AI-generated text.¹¹ The authors of this paper also played a pivotal role during the pandemic in identifying fake news and fake science, introducing a network-centric algorithm to address this issue.¹² Additionally, the authors of this paper tackled AI-generated text by introducing a groundbreaking algorithm trained on real scientific publications from the biomedical domain.¹³

Computational fact-checking vs. human verification

Although human verification of content is a much-needed task before establishing trust in AI-generated content, it is impossible for any human to cope with the massive volume and rapid pace of generation by such tools. What makes matters harder is the very convincing tone and appealing language used by all these tools. To demonstrate, we triggered ChatGPT, Bard, and Bing to generate their own versions of the article by Menczer et al.¹⁴ Specifically, the task was triggered using the exact title and specified the exact three anchors stated in their communication (detection, moderation, and regulation). The three documents resulting from this trivial exercise were strikingly convincing and hit many points right on target. Three instances included three paragraphs clearly labeled after the three sections. More impressively, they also overlapped in relevant terms such as “inauthentic content” and “social media”. Further, some of the tools used convincing terms such as “online platforms” and “spread misinformation” and sophisticated acronyms (“AI-generated inauthentic content [AIGC]”). It is impossible for any human to identify which one of the four documents was the authentic text of an original publication. While it is essential to verify the contents of generative AI tools, it is not feasible to declare human verification as a priority, as Van Dis et al. recommended.³ Instead, here we offer a viable and possible solution that overcomes the bottleneck of the human expertise needed. By leveraging the human knowledge that is already documented in knowledge bases, gold-standard databases, and domain-specific ontologies, we can computationally construct ground truth that can offer fact-checking capabilities for the necessary verification.

Transparency and explainability

Transformer-based methods suffer from what is known as the black box problem.³ This means that the results of such tools are not explainable and are sometimes hallucinatory. In the example of generative AI tools (e.g., ChatGPT, Google Bard), the end user can potentially receive different answers to the same questions without any justification, explanation, or references. In the effort of harnessing the potential of these tools, researchers must prioritize explainability and reproducibility while adhering to universal governmental orders (Biden’s executive order) and ethical standards.^15,16 In addition, we strongly recommend the incorporation of explainer-type algorithms that are associated with modern transformer-based and GNN approaches.^17,18,19 These types of explainable algorithms are becoming increasingly important as they provide the transparency and trust that are needed in the results, especially within the biomedical field. For instance, Pfeifer et al. developed an explainable GNN approach for identifying disease subnetworks.²⁰ This demonstrates the potential of these algorithms to enhance the transparency and trustworthiness of transformer-based AI models. By addressing these challenges, researchers can harness the potential of generative AI while ensuring that its outputs are transparent and ethical.

Research agenda

Back-to-basics

We propose a “back-to-basics” approach that utilizes traditional algorithms known for their inherent traceability. These algorithms have a proven track record of effectiveness. We refer specifically to the most fundamental data mining algorithms that are inherently traceable. This means that each step of the process can be consistently replicated, whether it is performed by a machine or a human. Our previous work in identifying the most fundamental and influential data mining algorithms is documented in a living reference known as the Top-10 Data Mining Algorithms.²¹ These algorithms can be employed for a wide range of computational tasks, including classification, clustering, ranking, boosting, association rule generation, and many others. To date, all of these algorithms remain widely taught in classrooms to students of all levels at numerous universities. They are also traceable, and their results are explainable. We advocate for this class of algorithms because their inherent traceability enables traversal mechanisms that facilitate the reproduction of results, both manually and programmatically. This class of algorithms is currently being investigated for detecting inauthentic content and fake science.¹³

Ontology-grounded truth

We believe that the integration of the biomedical ontology field is still in its early stages of development, and there is much work to be done to fully harness its potential in distinguishing between what is factual, what is hypothetical, and what is inauthentic, even if it appears credible. Additionally, exploring integrated ontology knowledge graphs adheres to the principles of network science and is therefore algorithmically traversable and computationally reproducible. Here, we present some examples of the scientific literature that has utilized biomedical ontologies as ground truth or empowered knowledge graphs that capture authentic knowledge using these ontologies.

Ontology-centric ground truth

The National Library of Medicine (NLM) often manually curates biomedical literature databases using ontology and taxonomy concepts due to their established authenticity.²² Notably, the NLM employs two prominent taxonomies: (a) the Medical Subject Headings (MeSH)²³ and (b) the NLM National Center of Biotechnology Information (NCBI) Taxonomy.^24,25 Following the NLM’s example, scientists have adopted ontology-based annotations as ground truth for identifying names and relationships in text.²⁶

Ontology-driven knowledge graphs

Blagec et al. developed an ontology-oriented knowledge graph to be used as a benchmark and for AI-specific tasks.²⁷ Ontologies have also been instrumental in powering knowledge graphs to build data-driven systems focused on safety information sharing.²⁸ Due to their reliability, ontologies have further enriched knowledge graphs with text features extracted from the literature, enabling reasoning and discoveries, as demonstrated by Chen et al..²⁹ Additionally, ontologies played a crucial role during the global pandemic in constructing credible knowledge graphs for accessing and retrieving knowledge related to the pandemic.³⁰ Clearly, the combination of ontologies and knowledge graphs presents a powerful approach for establishing ground truth and countering fake science.

The integration of various ontologies to form knowledge graphs in the biomedical field offers a promising platform for capturing precise facts and concrete relationships. Although these methods are still under investigation, they have the potential to provide a rigorous mathematical framework for distinguishing between factual content and the hallucinations generated by generative AI tools such as ChatGPT.³¹

Explainable AI

Despite the inherent lack of explainability in generative AI tools such as ChatGPT, we conducted a prompt-engineering experiment to investigate whether this aspect could be enabled on demand when generating a response. Specifically, we prompt-engineered ChatGPT to function as an “expert system shell” by encoding publicly available knowledge as structured rules, such as the “American Cancer Association screening recommendations for women at average breast cancer risk” available online.³² The prompt also required the request containing the response to provide an explanation of how the result was derived and which rules were triggered in making the recommendation. Figure 1 shows a screenshot of the ChatGPT prompt. The figure demonstrates how the prompt was engineered to guide the request and also demanded an explainable response that is derived solely from the entered rules and not from ChatGPT’s default pre-training process. When we put the rules to the test using a hypothetical case scenario, ChatGPT accurately assessed the situation, triggered the corresponding rules, and provided a detailed explanation of how the answer was derived. This experiment showcased not only the tool’s remarkable intelligence but also the significant role of prompt engineers in creatively utilizing generative AI responsibly. Figure 2 presents the hypothetical scenario and the specific rules that ChatGPT activated until the correct result emerged, accompanied by a comprehensive explanation based on the exact rules we entered and triggered accordingly. Here, we provide the entire prompt session, rules entered, and how they were invoked before the response was explained.³³ In the absence of such knowledge and structured rules, ChatGPT fell back on its pre-trained, unknown dataset, generating generic responses without providing any explanations. Figure 3 illustrates ChatGPT’s default response when lacking the rules and knowledge encoding, relying solely on its pre-trained model.

Figure 1.

Prompt-engineering ChatGPT to perform reasoning and produce explanation

Figure 2.

Prompt-engineering ChatGPT to perform reasoning and produce explanation

Figure 3.

Querying ChatGPT using its default pre-trained engine without encoding of knowledge into structured rules and not demanding explanations of how the answer is derived

With these experiments, we offer the following guidelines to advance the explainability aspect of the research agenda: (1) researchers and users of ChatGPT, Bard, and other generative AI tools should focus on the creativity of prompt engineering to insist on the explainability of such tools; (2) developers of ChatGPT and Google Bard should integrate built-in explainability mechanisms. Scientific workflows, like Pegasus WMS, serve as examples of how explainability can be achieved.³⁴ The generative AI industry, in particular, should advance the capabilities of such tools to include a referencing mechanism where the knowledge sources are appropriately cited; (3) researchers and users of transformer-oriented tools and methods (e.g., GNN) should seriously consider the use of explainable algorithms and actively contribute to advancing this research area^17,20; (4) during the peer-review process, reviewers should be vigilant in assessing the necessary transparency that fosters confidence in the results and demonstrates reproducibility, especially in research related to health and medicine; and (5) scientific journals and publishers should establish guidelines that elevate standards and mandate transparency and reproducibility in scientific research.

Policy priorities

This study addresses several pressing issues in the wake of emerging generative AI tools: (1) the imperative to identify inauthentic content; (2) performing fact-checking; and (3) the critical need for computational explainability and result verification. However, the current stage of AI technology development is associated with a number of ethical concerns, of which bias, privacy, accountability, and transparency are the leading ones. While technology development is fundamental to increasing the social benefits of this technology’s application, especially given the ongoing international competition for strategic innovation, the use of AI in scientific ecosystems has to be accompanied by widespread adoption of ethical standards. Otherwise, the uncertainty factor well identified with AI itself can easily be transmitted to the scientifically published research outcomes created with the use of technology.³⁵ Therefore, the approach proposed earlier goes far beyond the technical aspects of scientific verification infrastructure and should be seen in the broader context of the social reception of science. As a result, we call for the following essential policies to address both the technical and ethical issues associated with the emerging technologies of generative AI and LLMs.

The first policy, is leveraging both foundational algorithms and advanced explainer-type algorithms, specifically tailored for elucidating the inner workings of transformer-based tools. By utilizing computational detection methods and encouraging explainable AI, we strengthen our ability to distinguish real information from fake, creating a strong defense against the spread of false information.

The second policy, we highlight the significance of harnessing the rich knowledge of our knowledge bases, encompassing ontologies and gold standards. We advocate for seamless resource integration, the adoption of universally accepted sources of ground truth, and the innovation of automated fact-checking mechanisms. This concerted effort seeks to establish strong boundaries between reality and fiction, supporting the integrity of information disseminated through the utilization of generative AI tools.

The third policy, is for ethical AI is suggested where we seize the new opportunities ethically. Clearly, in the current era of AI and new advancement, a wide range of unprecedented opportunities emerge, alongside an array of hard challenges.³⁶ In order for the various communities to seize the opportunity, we must observe some of the foundational ethical standards outlined by Muller et al., who provide such aspects in their work titled “The Ten Commandments of Ethical Medical AI”.¹⁶ We hope that such policy will extend to the institutional level, where each organization mandates customized ethical standards pertaining to the specific purpose, research, and industry.

Conclusion

In this rapidly evolving research landscape, our recommendations described in the research agenda to support the computational detection of generated content, alongside computational fact-checking and explainability mechanisms, stand as a robust solution to counter the challenges posed by AI-generated tools and methodologies. Through the insights gained from our present work, and drawing from the experiences of the previous generation of conversational (non-generative) AI and chatbots,^11,37 we strive to safeguard authenticity while we foster positive experiences and establish trust. Our goal is to seize the opportunities presented by these powerful tools ethically and responsibly, marking an era characterized by the dissemination of true knowledge and reproducible science.