A conceptual framework for human–AI collaborative genome annotation

Abstract

Genome annotation is essential for understanding the functional elements within genomes. While automated methods are indispensable for processing large-scale genomic data, they often face challenges in accurately predicting gene structures and functions. Consequently, manual curation by domain experts remains crucial for validating and refining these predictions. These combined outcomes from automated tools and manual curation highlight the importance of integrating human expertise with artificial intelligence (AI) capabilities to improve both the accuracy and efficiency of genome annotation. However, the manual curation process is inherently labor-intensive and time-consuming, making it difficult to scale for large datasets. To address these challenges, we propose a conceptual framework, Human-AI Collaborative Genome Annotation (HAICoGA), that leverages the synergistic partnership between humans and AI to enhance human capabilities and accelerate the genome annotation process. Additionally, we explore the potential of integrating large language models into this framework to support and augment specific tasks. Finally, we discuss emerging challenges and outline open research questions to guide further exploration in this area.

Introduction

Genome annotation (GA) is the process of identifying and interpreting the functional elements encoded within a genome. It is a critical step in understanding an organism’s biology, enabling researchers to connect genetic information to phenotypes, understand disease mechanisms, and uncover evolutionary relationships. GA is heavily relies on automated methods, including machine learning (ML) [1–3] and other computational methods such as rule-based and heuristic methods [4, 5]. However, automated methods are generally hampered by the relative scarcity of reliable labeled data and the complexity of biological systems. In fact, gene annotations, particularly functional annotations, are mostly transferred from one species to another in an automated manner, relying mainly on the similarity of underlying nucleotide sequences, or the corresponding protein sequences.

Manual curation is widely recognized as essential for improving the reliability and accuracy of GA [6–8]. It involves human experts reviewing and refining annotations, particularly by addressing ambiguities or gaps that automated pipelines may overlook. For instance, curators enhance the functional annotations of genes by incorporating new insights from scientific literature that detail experimental results related to gene function. Additionally, manual curation enables precise gene structure annotation by reviewing evidence from multiple sources, such as omics datasets and experimental outcomes, to accurately define gene boundaries [9].

Despite their value, these evidence sources are often scattered across multiple platforms or embedded within vast datasets, making manual curation a time-consuming and labor-intensive process. Consequently, current GA practices rely heavily on automated annotation, which is not always followed by manual curation [10].

Manual curation has mostly been conducted in cases where teams of annotators collaborate to create accurate and up-to-date annotations for high-priority species or specific gene sets. Correspondingly, computational tools and platforms have been developed to support collaboration among annotators. These tools include algorithms that identify problematic annotations and prioritize them for review [11], as well as platforms that facilitate seamless communication, data sharing, and coordination among annotators, regardless of their geographic location [12, 13]. However, these tools operate independently and have not addressed the issue of dispersed data sources across various platforms. Additionally, there is a lack of dynamic interaction between the tools and users, i.e. the tools typically run automatically without a user in the loop.

As highlighted by Mac et al. [14], the disconnect between artificial intelligence (AI) tools and their users can potentially impact the effective utilization of AI. They suggest integrating scientists into the loop and combining their expertise with interactive ML to accelerate scientific progress. There is a growing body of research on how humans and AI collaborate to drive advancements in fields such as medicine [15–17] and chemistry [18], particularly through the adoption of large language model (LLMs) (see LLM in the Glossary section). Furthermore, the emerging concept of human–AI collaborative intelligence focus on the combination of humans and AIs working together to solve problems, leveraging the strengths of both parties and enhancing each other’s capabilities [19, 20]. Although still in its early stages, an increasing number of studies demonstrate that human–AI collaboration (HAIC) can lead to superior performance in accomplishing complex tasks [19, 21, 22].

Inspired by these work, we propose a conceptual framework named Human-AI Collaborative Genome Annotation (HAICoGA), in which humans and AI systems not only work interdependently but also collaborate over a sustained period. Collaboration in this context refers to effective functional integration between humans and AI systems. AI systems generate annotation suggestions by leveraging automated GA tools and relevant resources, while human experts review and refine these suggestions to ensure alignment with biological context and domain knowledge. This process is not a one-time interaction but an iterative collaboration, in which humans and AI systems continuously inform each other, enhancing both the accuracy and usability of AI support tools. Facilitating this collaboration is the use of LLM-based agentic systems, which integrate multiple tools and resources into a unified, interactive platform, streamlining the GA workflow and reducing the burden of tool switching or manual integration.

The remainder of this paper is organized as follows. Section 2 provides the necessary background and reviews related work relevant to this study. Section 3 introduces the conceptual framework of HAICoGA, identifying key components and critical capabilities required to establish an effective and sustainable human–AI collaborative relationship. In Section 4, we explore current applications of LLM-based AI agents in the biological and biomedical domains and present a vision for the HAICoGA workflow. Section 5 outlines key future research directions to further realize HAICoGA. We hope this work contributes to the development of human–AI collaborative workflows for GA in the future.

Background and related work

Genome annotation

GA can be interpreted as multidimensional, spanning from the nucleotide level to the biological system level [23]. Genomic elements of interest include, but are not limited to, single nucleotide polymorphisms, coding genes, noncoding genes, regulatory elements, and other noncoding regions. Structural annotation primarily focuses on delineating the physical regions of genomic elements. While the structural annotation offers initial clues, a definitive understanding of functions still requires in-depth analysis.

GA encompasses a broad range of tasks that are now primarily accomplished through various computational approaches utilizing diverse data types. We provide a rough chronology of the emergence and prominence of different automated methods in Supplementary Note 1. These automated methods can be integrated into highly complex pipelines to perform multiple steps in automated GA. Although automated approaches dominate GA, they still face serious limitations and challenges (Supplementary Note 1).

Due to the limitations of current computational tools, automated GA frequently produces erroneous results. In particular, genes from non-model organisms are often assigned functions based on homologs or labeled with vague terms such as “hypothetical gene” or “expressed protein,” offering little insight into their biological roles. These inaccuracies not only affect immediate interpretations but also propagate through downstream analyses [24], where they can be further amplified by ML or AI models trained on these data [25]. This creates a feedback loop in which low-quality annotations degrade the reliability of both current databases and future research that depends on them.

Manual curation

Manual curation has primarily been done in model species to continuously improve the accuracy and coverage of their GAs. For example, projects such as HAVANA for the human genome, TIGR for Arabidopsis thaliana, and ITAG for Solanum lycopersicum produce high-quality annotations manually curated by specialized experts. Manual curation is not limited to collaborative efforts or decentralized networks (detailed manual curation models are provided in Supplementary Note 2); it also plays a crucial role in individual research. Researchers may engage in manual curation before formulating hypotheses for their studies or when interpreting their results.

Manual curation is an ongoing process that requires the continuous repetition of five general steps [26, 27] (see Supplementary Note 2). This process is time- and labor-intensive, but it can be made more efficient with the assistance of software tools.

For example, Apollo has been widely used in the GA community and continues to be actively updated [12, 28, 29]. Its current web-based interface supports real-time collaborative annotation and integrates JBrowse [30] for fast, scalable genome visualization. Within Apollo, users can edit gene models, adding or deleting exons, adjusting boundaries, and assigning annotations. But functional annotations still rely on external tools, such as text mining systems. Tools like PubTator Center [31] assist in extracting biological entities and gene functions from literature, yet manual curation remains challenging. This is evident in efforts like BioCreative IV [32], which highlight the expertise and validation required to curate functions even with automated support [33].

More details on additional tools can be found in Supplementary Note 2. These tools are distributed across different platforms. Additionally, automated GA tools operate independently from these manual curation tools. As a result, humans need to spend a significant amount of time running and navigating multiple tools, as well as transferring data between them. To accelerate the GA process, an integrated system is needed, which connects all necessary automated and manual GA tools and enables seamless collaboration between humans and AI tools.

Human–AI collaboration

AI systems can play different roles in human–AI teaming, including automation, augmentation, and collaboration [34]. In automation, AI independently performs tasks without human intervention. In augmentation, AI enhances human experts’ abilities in their tasks. Collaboration refers to humans and AI working together in a coordinated effort toward overall goals, enabling better outcomes than either could achieve alone. Similarly, Gao et al. [15] classify AI systems into four intelligence levels based on their capabilities in hypothesis generation, experimentation, and reasoning. Level 0 consists of AI models used by humans as automated tools. Level 1 includes AI assistants that execute tasks specified by scientists. Level 2 consists of AI collaborators that work alongside scientists to refine hypotheses and utilize a broader array of tools for experimentation and discovery. Level 3 represents AI scientists, which exhibit the highest level of intelligence.

In the current GA workflow, automated GA tools can be categorized as Level 0 AI models. Recently, LLMs have been used to develop various AI assistants (Level 1) in the biological and biomedical domains, which could potentially be adapted for GA. However, a significant gap remains due to the lack of HAIC in GA (Level 2).

The emergence of HAIC frameworks and taxonomies across various domains has advanced our understanding of effective HAIC. For instance, Hartikainen et al. [35] propose a framework for designing collaborative systems in smart manufacturing, while Martin et al. [36] present a model for human–AI co-design in Design Space Exploration. Dubey et al. [37] introduce a framework for successful human–AI teaming in contact centers. Despite this progress, applying HAIC to GA presents unique challenges. We build on these frameworks to introduce HAICoGA and propose future research directions for collaborative GA.

A conceptual framework of human–AI collaborative GA

To support collaborative GA, HAICoGA incorporates and extends key concepts from general HAIC theories [35, 37, 38]. As shown in Fig. 1, it includes seven key elements: humans, AI systems and tools, data, goals and tasks, the human–machine interface, environment, and collaboration. Humans and AI operate as a collaborative team within a dynamic environment. Tasks and goals are continuously updated as they interact via a shared interface, leveraging available data to achieve genome annotation outcomes.

Figure 1.

Key elements in human–AI collaboration for genome annotation.

Key elements

Humans

Humans refers to individuals involved in GA, particularly those engaged in manual curation. This group may include biological researchers, experimental scientists, biocurators, and trained students. These individuals bring diverse backgrounds, domain knowledge, and methodological experience to the decision-making process [39]. Research suggests experts rely heavily on their previous experience on the task, as well as their deep and often tacit knowledge of the domain [40]. Decision-makers often employ mental shortcuts and heuristics that are efficient but can be prone to cognitive bias [41, 42].

Collaborating with LLMs has the potential to mitigate individual biases when the complementary strengths of human judgment and machine predictions are effectively leveraged [43]. Still, bias in human curation remains an open and ongoing challenge in the biological domain. For example, the BC4GO study [44] demonstrated substantial variability in manual GO annotation, with low inter-annotator agreement even among trained curators, highlighting that human bias persists regardless of AI involvement and must be continually recognized and mitigated.

While humans may not match AI in processing speed or handling vast datasets, they possess the ability to interpret nuanced context, adapt to shifting task goals, and act flexibly in dynamic environments [40]. This adaptability is crucial in GA, where the context and requirements may change based on new findings or experimental results. Humans can also perform sanity checks and intervene when AI-generated annotations deviate from established rules or expectations, helping to limit the propagation of errors throughout the system [45]. Consequently, while AI can assist and augment decision-making, the role of experienced human curators remains indispensable [46].

AI systems and tools

The AI element consists of a collection of systems and tools designed to perform or assist in GA. By using the term “AI,” we encompass a broad range of computational tools that can be integrated into intelligent systems to enhance collaboration with humans during GA tasks. Ideally, the AI component should consist of an ecosystem of tools with diverse designs and functionalities, each contributing uniquely to the annotation process.

Automation-focused AI tools streamline the multi-step process of identifying and classifying genes and other functional genomic elements, as seen in established GA pipelines and automation methods [5, 47]. In contrast, augmentation tools are designed to assist human annotators by enhancing their efficiency and effectiveness, for example, Apollo [12]. Collaboration tools could support bidirectional interaction between humans and AI systems.

While certain biases inherent in AI systems (discussed in Supplementary Note 1) cannot be eliminated, they can be mitigated through “the wisdom of the crowd” approach that combines the complementary strengths of both humans and diverse AI systems.

Data

HAICoGA involves two main types of data: those used for GA and those that support HAIC. Genomic sequence data play an important role in GA, including genome assemblies, expressed sequence tags (ESTs), complementary DNAs (cDNAs), RNA-seq data, and protein sequences [10]. Biological databases, such as UniProt [48] and Gene Ontology (GO) [49], aggregate knowledge from experiments and studies, providing labeled data for GA. Knowledge graphs (KGs) further enrich GA by integrating heterogeneous biological data into structured formats [50]. Scientific publications serve as contextual evidence for gene annotations. Grounding annotations in traceable literature evidence ensures that final annotations are supported by verifiable sources.

Collaboration-related data capture how users engage with AI systems, for instance, the queries they enter, options they choose, and feedback they provide. These data help refine AI algorithms to better suit user needs.

Goals and tasks

Recent research in HAIC highlights that effective teamwork involves aligning AI behavior with human objectives, enabling both to contribute toward common goals [51]. These goals can be achieved through structured plans composed of subtasks. In hierarchical task analysis, a task is broken down into subtasks until a stop criterion is reached, often when the subtask consists of only a single operation [52]. For example, a single operation such as gene prediction may be performed by an AI tool, while others may be handled by humans, such as reviewing predictions. Plans define the sequence and structure of tasks, whether sequential or hierarchical, and help allocate specific subtasks to either human experts or AI systems [52].

Human–machine interface

The human–machine interface serves as the bridge enabling interaction between humans and AI systems. It includes various forms of interfaces, including command-line interfaces (CLIs), graphical user interfaces (GUIs), and conversational user interfaces (CUIs).

CLIs provide users with greater flexibility through programmatic access, customization options, and the ability to perform batch operations or access raw data directly. However, they can be challenging for users without programming experience. In contrast, GUIs provide a more user-friendly experience by allowing users to interact with AI systems through visual elements. GUIs are widely adopted in GA tools for tasks like visualizing feature locations, displaying evidence alignments, and presenting other relevant information. Genome browsers such as JBrowse [30] provide graphical interfaces for viewing GAs alongside supporting evidence tracks, while annotation platforms like Apollo [12] extend this functionality by enabling users to collaboratively edit and curate the annotations.

While GUIs are indispensable for the curation of structural annotations, CUIs backed by tool-wielding AI agents have the potential to fulfill an analogous role in functional annotation. Recent advancements in LLMs have significantly contributed to the growing interest in CUIs. CUIs allow users to interact with machines using natural language, making it easier to access information and perform tasks without needing to memorize complex commands or navigate intricate menus. For certain tasks, CUIs enhance HAIC by enabling effective and intuitive interactions between humans and AI [53].

Environment

Environment plays a critical role in shaping human perceptions of AI, influencing interaction dynamics, and affecting the AI system’s capacity to interpret and respond to human input. It can be broadly categorized into digital, task, and team environments. The digital environment includes conditions and factors such as software platforms, interface design, and the availability of data for both humans and AI [54]. The task environment pertains to the tasks that need to be completed, the constraints and limitations involved, and the desired outcomes [55]. The team environment refers to the dynamics and structures within a group of individuals (including both human and AI) working together [56]. It is characterized by the roles and relationships established among team members, communication patterns, and the level of cooperation and collaboration required to achieve overall goals.

Collaboration

Whether human–AI interaction constitutes true collaboration remains a subject of ongoing debate [57, 58]. In the HAICoGA framework, we use the term collaboration to refer to a structured interaction between human experts and AI systems, wherein complementary strengths are integrated to improve GA workflows. Effective HAIC relies on meaningful interaction and strategic alignment of distinct capabilities [20, 59]. Understanding these respective strengths is essential for designing effective collaborative systems in GA (Fig. 2; Supplementary Notes 3 and 4).

Figure 2.

Key competencies for effective HAIC. Human competencies include learning, reasoning, situational awareness, decision-making, delegation, and trust. AI competencies mirror and complement these with learning, reasoning, contextual awareness, prediction, deferral, and explainability. Co-learning supports mutual adaptation. The dashed line represents an iterative feedback loop between humans and AI. See Supplementary Note 4 for details.

Human capabilities include abstract reasoning, situational awareness, and nuanced decision-making—capabilities that AI does not inherently possess, but may emulate to a limited extent through task conditioning and model fine-tuning. In contrast, AI offers scalable computation, rapid pattern recognition, and the efficient execution of repetitive or high-volume tasks such as candidate gene prioritization and evidence retrieval.

While automated GA tools can be error-prone, collaboration between humans and AI can improve the accuracy and reliability of the final outputs. AI systems should be designed to recognize tasks that exceed their confidence thresholds, such as annotating short genes or resolving complex duplications, and defer these cases to human curators for review. In turn, humans provide feedback by correcting errors, validating results, or suggesting alternative approaches. AI systems can then incorporate this feedback to refine their outputs, aligning more closely with human expectations and the dynamic context of the annotation process. This adaptive capability is referred to as contextual awareness. Bidirectional feedback mechanisms of this kind help prevent erroneous predictions and reduces the risk of propagating errors in GAs. Moreover, ongoing efforts to improve AI explainability are critical to fostering effective collaboration, as they enhance transparency and build trust in automated systems.

AI agents bring opportunities to realize the HAICoGA framework

L‌LM-based AI assistants in biological and biomedical domains

There is an emerging trend of using LLM-backed AI assistants for research in biological and biomedical domains. These AI assistants can process human language inputs and generate responses that are coherent and contextually relevant within the interaction. We categorize these research based on the number of agents involved (Table 1). An agent refers to an AI system capable of interacting with humans or other agents and using tools to accomplish its tasks.

Table 1.

AI assistants in biological and biomedical domains

Method	Number of agents	Data	Task and goal	Team structure	Tool use	Explainability
ChatNT [63]	No agent	DNA, RNA, protein sequences and text data	Interpret biological information encoded in genome sequences and provide accurate predictions for various biological functions, such as gene expression prediction, DNA methylation, RNA stability, and protein properties.	NA	NA	NA
DRAGON-AI [64]	No agent	Structured data from existing ontologies and unstructured textual data from sources like GitHub issues	Generate ontological terms.	NA	NA	NA
GeneGPT [65]	No agent	Text data	Answer genomics-related questions by directly generating API request URLs to access and retrieve relevant biomedical information.	NA	NA	NA
Phenomics Assistant [66]	Single AI agent	Monarch knowledge graph	Enhance accessibility to complex genomic information by enabling natural language querying of the Monarch knowledge graph.	NA	Monarch Initiative API	The explainability of AI-generated answers by grounding them in data retrieved from the Monarch KG.
VarChat [67]	Single AI agent	Scientific literature and human genomic variants	Support genetic professionals by providing concise summaries of scientific literature related to specific genomic variants.	NA	Query genomic databases; find and summarize the fragmented scientific literature	Informing users about the sources of its responses.
ChatGSE/ biochatter [68]	Two AI agents	Knowledge graph and scientific articles	Answer user’s questions using context from knowledge graphs and scientific papers; demonstrate the usability in cell type annotation task.	Sequential	Information retrieval from knowledge graphs and the literature	Fact-checked and supplemented with context-specific information from documented sources.
BioDiscoveryAgent [69]	Two AI agents	Biological database (Reactome 2022 database) and literature	Design genetic perturbation experiments that efficiently navigate the hypothesis space to identify a small subset of genes resulting in specific phenotypes.	Sequential	Search the biomedical literature and execute code to analyze biological datasets	Detailed explanations for its choices, including citing relevant literature and detailing the reasoning behind selecting specific genes for perturbation.
GeneAgent [70]	Two AI agents	GO, Molecular Signature Database (MSigDB), and a proteomics analysis system (NeST)	Generate biological process names for gene sets.	Sequential	Call web APIs that connect to biological databases	Providing verification reports that detail the evidence supporting or refuting each generated name.
BRAD [71]	Multiple AI agents	Online literature repositories, Enrichr and Gene Ontology databases	Automate bioinformatics workflows, enhancing the speed and efficacy of tasks such as gene enrichment analysis, literature searching, and running software pipelines.	Hierarchical	Search online literature, execute code to run software pipelines, such as enrichment analyze and visualization	Providing context-rich answers that include references to data sources and literature.
BKGAgent [72]	Multiple AI agents	Clinical Knowledge and academic literature Graph	The primary task is Knowledge Graph Checking, which involves querying KGs, verifying the correctness of the information using external literature or databases, and identifying factual errors.	Hierarchical	Specific tools for interacting with knowledge graphs and scientific literature	Agent actions and decisions are traceable and justifiable, particularly in the context of verifying scientific claims and correcting knowledge graph data.
GenoAgent [73]	Multiple AI agents	GEO and TCGA databases	Automate the analysis of gene expression data to identify disease-associated genes.	Hierarchical	Various bioinformatics tools, such as those for data normalization and statistical analysis	Provide explainable results by documenting the decision-making process and the steps followed in the data analysis.
ProtAgents [74]	Multiple AI agents	Protein sequences, structural data, simulations and external databases	Automate and enhance the design of novel proteins with specific mechanical properties. This involves generating new proteins, analyzing their structures, and obtaining new first-principles data through physics simulations.	Hierarchical (dynamic, collaborative multi-agent environment)	Physics simulators and generative AI models, to perform tasks ranging from data retrieval to complex simulations of protein behaviors.	Provide explainable results by detailing the reasoning behind its decisions, the data used, and the methodologies applied.
[75]	Multiple AI agents	Single-cell RNA sequencing (scRNA-seq) data and literature	Replicate the experimental and analysis process of a scientific publication that explored gene expression relevant to SARS-CoV-2 entry into human cells. The goal is to validate the methods used in the original publication and to enhance the reproducibility and transparency of scientific research using AI.	Hierarchical	Software tools for data analysis and paper summary	Providing detailed breakdowns of its analytical processes and how decisions and analyses are derived, allowing for a transparent review of its methodology replication.
TAIS [76]	Multiple AI agents	TCGA, NCBI Gene and GEO databases	Identify disease-predictive genes from gene expression data.	Hierarchical	Various computational tools and methods integrated into the data processing and analysis workflows	Not detailed in the paper.
Virtual Lab [77]	Multiple AI agents	Public protein databases and SARS-CoV-2 variant data	Design and validate nanobody binders for SARS-CoV-2 variants using AI-driven workflows.	Hierarchical	Bioinformatics tools for protein analysis	Providing explainability by structuring AI-agent meetings, documenting decisions, and presenting clear computational workflows.

NA: not applicable. https://monarchinitiative.org/. https://reactome.org/. https://www.gsea-msigdb.org/gsea/msigdb. https://idekerlab.ucsd.edu/nest/. https://maayanlab.cloud/Enrichr/. https://www.ncbi.nlm.nih.gov/geo/. https://www.cancer.gov/ccg/research/genome-sequencing/tcga

Some works in Table 1, such as ChatNT, DRAGON-AI, and GeneGPT, are LLM-based models that take human language as input and generate direct answers without involving a agent. ChatNT is a multimodal AI system that integrates DNA, RNA, and protein sequences with neural language processing to solve various genomics tasks. ChatNT employs a modular LLM architecture that integrates a bidirectional DNA encoder with a unidirectional language decoder, enabling the interpretation of biological sequences in natural language. Similar architectures are discussed by Zhang et al. [60], who compare unidirectional and bidirectional LLMs in biological and chemical domains. Bidirectional models (e.g. BERT [61]) excel at encoding sequences for classification tasks, while unidirectional models (e.g. GPT [62]) are well-suited for generative tasks like text generation.

DRAGON-AI is a method that automatically generates ontology objects based on partial information from a user. All ontology terms and additional contextual information are translated into vector embeddings and indexed. Relevant contextual information is retrieved using a retrieval-augmented generation (RAG) approach and added to construct a prompt, which is then passed as input to an LLM. The LLM completes the term object accordingly. GeneGPT uses few-shot learning to teach LLMs how to generate web APIs for accessing the National Center for Biotechnology Information (NCBI) databases and to answer biological questions based on the retrieved information. It handles both single-hop questions, which require a single API call, and more complex multi-hop questions that necessitate sequential API calls. For multi-hop questions, GeneGPT decomposes them into sub-questions, executing a chain of API calls to retrieve and integrate information step by step. While GeneGPT automates this process, its authors acknowledge that different types of errors are enriched in different tasks. Given the complexity inherent in multi-hop reasoning, such cases may still benefit from human oversight to ensure the accuracy and reliability of the results.

Phenomics Assistant builds an agent to call external tools based on user queries. It helps non-expert users query and interact with complex data from the Monarch Knowledge Graph. VarChat supports genetic professionals by providing concise summaries of scientific literature related to specific genomic variants. It interacts with external databases and utilizes user inputs to guide its querying and summarization processes. Both Phenomics Assistant and VarChat use a single-agent framework to provide a CUI that interacts with users and has the ability to use tools to solve user questions based on dynamic situations. The conversation history in the chat allows the agent to be aware of the user’s state within tasks and incorporate feedback from external tools. Both systems also provide sources for the information in their responses, improving transparency in their processes.

Two-agent systems, ChatGSE, BioDiscoveryAgent, and GeneAgent, consist of a primary agent that interprets the user’s query and selects appropriate tools to solve the problem, and a secondary agent that critically evaluates the results or verifies the factual accuracy of the output. The tools either retrieve and process information from various APIs to access online databases or scientific literature. The retrieved information is treated as a source to determine whether the answer is factually accurate compared with the original data. Keeping track of intermediate results from tools and the verification process enhances the agent’s awareness of the current task status, potentially allowing it to adjust its actions accordingly in the next round of experiments. ChatGSE employs chain-of-thought reasoning to improve its problem-solving success. BioDiscoveryAgent follows the Reflection-Research Plan-Solution framework to enhance its reasoning capabilities. Both ChatGSE and BioDiscoveryAgent also incorporate self-verification mechanisms. These two agents operate in a sequential manner. All three systems provide some level of explainability by delivering context-rich answers that include references to data sources, literature, or verification reports. GeneAgent, which applies an AI agent for gene set enrichment analysis, focuses on autonomous interactions with domain-specific databases, followed by subsequent LLM verification.

Multi-agent systems are becoming increasingly popular for solving complex problems. These systems integrate multiple AI agents to automate and enhance critical workflows, significantly improving the speed and efficacy of tasks such as gene enrichment analysis, literature searches, and software pipeline executions. For instance, the BRAD system employs a hierarchical structure of agents to manage tasks like literature retrieval and enrichment analysis automation. These agents use a combination of in-context learning and a specialized planner to distribute and organize tasks efficiently. Another example is the BKGAgent, which focuses on knowledge graph checking by querying knowledge graphs, verifying the accuracy of information through external literature or databases, and identifying factual discrepancies. The system’s ability to dynamically query and cross-reference structured knowledge graphs and unstructured scientific texts illustrates the integration of RAG, ensuring relevance and contextual awareness throughout the information processing stages.

Similarly, GenoAgent and TAIS are tailored for analyzing gene expression data from sources like the Gene Expression Omnibus (GEO) and The Cancer Genome Atlas (TCGA). These systems leverage instruction learning and structured prompting to adapt their actions based on feedback and intermediate results, facilitating an iterative correction process that ensures the reliability and explainability of analytical outputs.

Beyond genomics, Virtual Lab exemplifies the application of multi-agent AI systems in experimental biomedical research. This system utilizes an AI-driven research framework, where a Principal Investigator AI leads a team of specialized agent, including a ML specialist, immunologist, and computational biologist, to design and validate nanobody binders for SARS-CoV-2 variants. The system’s ability to document decision-making steps and optimize AI-driven workflows highlights the growing role of multi-agent systems in interdisciplinary research.

Lastly, ProtAgents showcases a multi-agent application in the design and analysis of novel proteins. By integrating real-time data from experiments and simulations, these agents can generate and analyze new proteins, adjusting their outputs based on dynamic inputs. The multi-agent system developed by Bersenev et al. [75] facilitates the replication of high-impact scientific studies by processing research papers and generating code to reproduce experiments, streamlining experimental validation and iterative scientific discovery.

Table 1 summarizes information from these studies, aligning certain elements with the HAIGoGA framework, including data, tasks, goals, AI systems and tools, and team structure (environment). The data, tasks, goals, and tools are customized for different AI assistants. In studies involving multiple agents, these agents are often organized hierarchically, with a high-level agent (e.g. planner, leader, or manager) responsible for task distribution and coordination of the analysis process. Regarding the human–machine interface, three studies provide both GUI and CUI to facilitate human interaction with AI agents [66–68]. The most recent work, Virtual Lab [77], demonstrates the impact of HAIC through experiential evidence. In this framework, agents can defer tasks to other agents, as well as humans.

Cognitive functions, such as perception, reasoning, planning, and memory, are essential for enabling LLM-based agents to maintain contextual awareness and generate relevant responses in human–AI interactions. For example, the ReAct agent integrates reasoning and action, iteratively repeating this process until it determines a final response. The agent evaluates the current input along with past observations to decide the next step [78]. Some AI systems incorporate memory management to continuously track user interactions and dynamically recalibrate the agent’s actions based on intermediate results and feedback [66–68, 71]. Table 1 shows that most studies support explainability through tracing agent actions, predictions, and the external data sources used.

Vision for the HAICoGA framework

Multi-agent system design in the HAICoGA framework

Through our review of current LLM agents in the biological and biomedical domains, we identified multi-agent systems as a promising approach for realizing the HAICoGA framework. Existing research primarily focuses on developing autonomous systems that minimize or even eliminate human intervention. However, such fully autonomous systems have demonstrated limited effectiveness in real-world applications [77, 79]. It is essential to keep humans in the loop to enhance system performance and reliability [18, 77].

Figure 3A illustrates an example of users collaborating with a multi-agent system to annotate gene functions. Based on the user’s input query, the manager agent could use a method (e.g. ReAct) for breaking down the query into subtasks and assigns them to other agents according to their capabilities (Fig. 3C and D). The critique agent evaluates the quality of task results using metrics such as completeness, relevance, and other task-specific criteria, providing feedback and indicating the task’s status. If necessary, agents can request additional input from the user. Once all tasks are completed, the manager agent compiles the final response and presents it to the user.

Figure 3.

Multi-agent system design in the HAICoGA framework. (A) Overall multi-agent system design for human–AI collaborative GA. Users submit a GA query through an interactive user interface (UI). The UI requests the manager agent to analyze the task, decompose it into subtasks, and assign them to appropriate agents. While assisting with a subtask, an agent may request additional input from the user to complete the task successfully. The critique agent provides feedback on the outcomes, guiding the system’s next steps. The manager agent monitors the global conversation history and intermediate results, updating the task plan as needed or finalizing the task and delivering the results to the user. (B) The top synergy layer of the multi-agent system designed for HAICoGA. Following the practical GA workflow [10], the multi-agent system consists of a user, a manager agent, an automated GA agent, multiple manual curation agents, and a critique agent. (C) Workflow of multi-agent collaboration in automated GA phase. The manager agent delegates the automated GA task to the automated GA agent, which manages a customized pipeline (or an AI model) using genome data to perform specific tasks. The critique agent analyzes the results, evaluates their quality, and suggests the next steps to the manager agent. This process can be repeated iteratively until the desired outcome is achieved. (D) Workflow of multi-agent collaboration in manual curation phase. A manual annotation process follows the automated GA phase. Due to the complexity of manual curation, the system includes several specialized agents performing distinct roles. The sequence search agent identifies homologous genes for a target gene, for example, by running BLAST against genome sequence data. The database agent retrieves gene function annotations from various databases. The literature search agent identifies relevant scientific papers for further analysis, while the document summarization agent extracts key information from these papers. The synthesis agent compiles all relevant data and submits it to the critique agent, which reviews the information and provides suggestions, such as whether the data is sufficient to address the user’s query. Finally, the manager agent either updates the task plan or generates the final response.

Building on the GA workflow described in the review by Ejigu et al. [10], we propose an automated GA agent along with several agents for manual annotation (categorized as manual curation agents in Fig. 3B, each assigned distinct roles, as detailed in Fig. 3D). While manual annotation is often performed based on the results of automated GA, newly added manual annotations can also enhance the automated GA system by providing additional gold-standard data, enabling continuous refinement of gene annotations.

Another key strength of multi-agent systems is that it allows for the internal refinement of answers. In the automated GA phase (Fig. 3C), the automated GA agent executes AI models and pipelines to perform specific tasks using genome data, such as predicting gene functions. The manager agent and critique agent contribute by summarizing results and providing feedback to the automated GA agent, which may prompt it to select alternative models or pipelines for gene function prediction. This iterative process enhances the quality of gene annotation. The self-improving loop continues until either the user or the manager agent decides to finalize the process and provide the final answer for the task.

The use of multiple agents also allows for specialization in the manual annotation system (Fig. 3D), each assigned distinct attributes, including role, perception, and actions (tool use). These attributes enable agents to be optimized for specific domains or functions [80]. To manually annotate an uncharacterized gene, several guidelines recommend a workflow that involves using a tool (e.g. BLAST) to identify homologous proteins, retrieving functional annotations from existing databases and recent literature, and assigning these functions to the target protein [7, 33, 81]. Following these guidelines, the manager agent is responsible for designing this workflow and distributing tasks among specialized agents, including the sequence search agent, database agent, literature search agent, and document summary agent. The synthesis agent then aggregates the results, while the critique agent evaluates the output and provides feedback to the manager agent. Similar to the automated GA phase, the user could interrogate the results and refine prompts to continuously refine the quality of gene annotation.

Illustrative use cases of the HAICoGA framework

To demonstrate the practical use cases of the HAICoGA framework, we highlight the application of the GeneWhisperer system for gene annotation [82].

GeneWhisperer employs an LLM agent integrated with domain-specific tools to assist in generating functional hypotheses for genes, particularly uncharacterized genes in a reference genome. The system synthesizes multiple forms of evidence by identifying homologous proteins through sequence alignment, proposing relevant Gene Ontology (GO) terms based on functional similarity, and extracting gene-trait associations from scientific literature.

Following AI-assisted annotation, domain experts would review the generated hypotheses, validating them against species-specific literature and related annotations in other genomes. While experts do not generate annotations entirely from scratch, they are able to refine, correct, or reject AI-suggested annotations based on domain knowledge. As noted by Kudiabor et al. [83], AI-assisted annotations, particularly for novel genes, should not be considered definitive without supporting wet-lab experiments. Furthermore, we acknowledge that for certain genes, neither the user nor the AI system may be able to produce a meaningful annotation when no relevant information currently exists.

Another use case of the HAICoGA framework involves an AI assistant designed to improve consistency in gene function annotation. Manual curation often results in variability due to the difficulty in selecting standardized GO terms and corresponding Evidence and Conclusion Ontology (ECO) codes.

The AI assistant would analyze user-provided inputs, e.g. literature excerpts, and suggests appropriate GO and ECO terms. Users would review and refine these suggestions, maintaining expert oversight throughout the process. We demonstrated an examples using ChatGPT as an LLM-based agent to assist in selecting GO and ECO terms (see Supplementary Note 5). While preliminary, this example illustrates the potential of general-purpose LLMs like ChatGPT can serve as accessible annotation assistants. It also highlights the limitations of such models in domain-specific tasks, underscoring the need for future development of specialized AI assistants built on the HAICoGA framework.

These illustrative use cases demonstrate the practical viability of the HAICoGA framework in supporting GA tasks through synergistic human–AI workflows. Similar ideas have been implemented in other scientific domains. For example, the AI Co-Scientist system leverages a multi-agent architecture to collaborate with scientists in hypothesis generation, drug repurposing, and biomedical discovery [84]. This iterative collaboration between AI systems and domain experts reflects the same core principles underpinning HAICoGA.

By optimizing agents for specific annotation tasks and integrating expert feedback, HAICoGA aims to extend these advances into the genomics space. In the following section, we discuss the remaining challenges and technical considerations in building such systems.

Challenges in building the HAICoGA framework

Designing the architectural of a multi-agent system

The design of LLM-based multi-agent systems requires a modular and adaptive architecture in which specialized agents collaborate dynamically through structured interaction layers. These agents, each with distinct roles, leverage LLM capabilities for reasoning and task execution while interoperating with external resources such as datasets and tools to maintain contextual awareness. Achieving this requires balancing autonomy and alignment, as excessive autonomy may lead to goal deviations, whereas strict alignment can hinder adaptability [85]. Furthermore, managing dependencies among agents and ensuring scalability in resource usage are critical, especially as tasks grow more complex. Mechanisms for real-time adaptation and error correction are also essential to address inconsistencies and ensure robust, goal-oriented outcomes in complex environments. Finally, challenges remain in optimizing task allocation, fostering robust reasoning through iterative debates, managing complex contextual information, and enhancing memory management [86].

Developing novel ML/AI methods for enhancing HAIC

LLM agents, particularly unidirectional models, facilitate dynamic communication with users, but recent research highlights several critical challenges that may affect their collaborative effectiveness. Hallucination remains a significant concern, in which models generate plausible-sounding but factually unsupported content [87]. As such outputs can influence decision-making, they risk propagating false beliefs or even causing harm, underscoring the need for robust mitigation strategies. Fine-tuning bidirectional models with sufficient domain-specific training data can significantly improve their performance in tasks such as information extraction and classification. To further enhance reliability, systems could support continuous learning, enabling dynamic updates through human feedback and evolving contexts, as exemplified by reinforcement learning from human feedback [80].

Maintaining context over extended interactions is another area where LLMs often falter, leading to incoherent responses or an inability to recall previous discussions. Vector databases offer a potential solution by enabling long-term memory management in LLM agents, allowing them to accumulate and organize memories over time. However, efficiently searching and retrieving relevant information from extensive memory stores remains challenging. Further advancements are needed to develop mechanisms for learning and updating metadata attributes across both procedural and semantic memory types [88]. MemGPT [89] exemplifies progress in this domain by intelligently managing different memory tiers to store and retrieve information effectively during long-term conversations.

Reasoning capabilities are pivotal for LLM agents to perform complex and nuanced tasks such as problem-solving, decision-making, and planning. Explicit reasoning steps not only improve task performance but also enhance model explainability and interpretability by providing rationales for predictions. While LLMs are primarily trained for next-token prediction, strategies like Chain of Thought (CoT) have demonstrated improvements in reasoning tasks by guiding models to articulate their reasoning explicitly. However, LLMs still face challenges in handling highly complex reasoning tasks or those involving subtle implicatures, necessitating ongoing research [90].

Requiring multidimensional evaluation methods to assess the HAICoGA workflow

Traditional GA evaluation metrics, such as coverage, precision, and accuracy, remain fundamental for assessing annotation quality [3, 91]. These measures indicate better outcomes when higher values are achieved; however, they provide relative rather than absolute benchmarks due to the absence of a comprehensive genome-wide gold standard. Many annotations remain provisional, relying on computational predictions or homologous transfers from model organisms.

In HAICoGA workflows, additional dimensions, such as explainability, are crucial for evaluation. Integrating orthologous information, along with detailed protein family and domain characterizations from diverse sources, enhances the explanatory depth and reliability of annotations [92]. Metrics that assess explanation generation and evidence quality are essential to ensuring the transparency of AI-assisted workflows. This aligns with frameworks for evaluating HAIC, which emphasize not only task success but also interaction quality, process dynamics, and ethical considerations [93].

Furthermore, optimizing the performance of human–AI teams requires a paradigm shift from individual AI optimization to assessing team-level outcomes. Evidence suggests that the most accurate AI system does not necessarily yield the best collaborative performance [94]. Effective collaboration depends on dynamic task allocation, mutual learning, and trust between human and AI agents. Metrics for evaluating such interactions must consider both qualitative factors, such as trust and satisfaction, and quantitative measures, such as decision impact and task completion time [93].

Adopting multi-dimensional evaluation frameworks, such as those emphasizing symbiotic HAIC modes, can provide holistic insights [93]. These frameworks should capture the dynamic, reciprocal nature of collaboration, extending beyond task success to evaluate how well humans and AI adapt to each other’s strengths and limitations over time. Such comprehensive approaches are crucial for advancing the HAICoGA workflow and ensuring its alignment with both scientific rigor and practical utility.

Designing intuitive and interactive interfaces to facilitate HAIC

To investigate the challenges and opportunities in CUIs, we developed a chatbot prototype for curating information in gene functional annotation [82]. Additionally, we proposed applying conjoint analysis, a behavioral science method, to quantify the relative importance of four design features that influence users’ trust in the system [95].

Initial testing of the prototype suggests that LLM agents have the potential to serve as valuable tools for collaborative GA when combined with human expertise. However, further research is needed to enhance their trustworthiness, particularly by improving explainability and providing confidence measures for AI-generated predictions [95].

To support these capabilities, future work will focus on integrating a dedicated graphical user interface (GUI) with the chatbot, particularly for structural annotation. Developing the right interface will be best served by taking a participatory or user-centered design approach and incorporating input from GA experts from the outset.

Risks and safeguards

The integration of LLM agents into scientific workflows introduces a set of risks that necessitate proactive and comprehensive mitigation strategies. The risks include the potential for generating misleading or harmful content, propagating biases, and compromising data privacy and security. These risks can arise from the inherent limitations of LLMs, such as their susceptibility to hallucination, over-reliance on training data, and the challenges of ensuring alignment with human values and ethical standards [96].

To mitigate these risks, a triadic framework involving human regulation, agent alignment, agent regulation and environmental feedback has been proposed [96]. Human regulation involves establishing clear guidelines and protocols for the responsible use and development of LLM agents in scientific contexts. This ensures ongoing human oversight and supports human-in-the-loop validation [97]. Agent alignment means that LLM agents are designed and trained to align with human intents and ethical standards, minimizing the risk of generating misleading or harmful content. Widely adopted safety mechanisms, such as those implemented in ChemCrow [18] and SafeScientist [98], can help ensure that agents operate within predefined boundaries and do not produce harmful outputs. Agent regulation and environmental feedback refer to the continuous monitoring and evaluation of LLM agent performance in real-world applications, enabling iterative refinement of their behavior. Feedback in multi-agent systems comes not only from human users but also from critique agents, external tools, and structured knowledge sources. Techniques like RAG exemplify how agents can be designed to incorporate trusted external knowledge sources, improving reliability and reducing the risk of hallucinated content [97].

Conclusion

In this paper, we first analyzed the pros and cons of automated GA methods and manual curation tools. We found that while automated GA methods generate annotations quickly, they have limitations, such as inaccurate gene predictions. On the other hand, manual curation can be highly accurate but requires intensive human labor and time. A human–AI collaborative GA approach is necessary to leverage the strengths of both humans and AI, leading to more accurate and efficient GA.

Bringing together prior work in automated GA and manual curation, we then proposed the conceptual framework of HAICoGA. Our work bridges the gap between GA and human–AI collaborative communities, envisioning new possibilities in this multidisciplinary field. The emergence of LLM agents presents significant opportunities to realize HAICoGA workflows. However, many challenges and open questions remain in LLM agent research. The HAICoGA framework is still in its early stages of development, but it represents a step toward a comprehensive and efficient human–AI collaborative workflow for real-world applications in the future.

Glossary

Genome annotation (GA) is the process of identifying and characterizing functional elements within a genome, including genes, regulatory regions, and other biologically significant sequences. It involves the use of computational methods, such as machine learning (ML) and heuristic-based approaches, as well as manual curation by experts to improve accuracy. GA is essential for understanding gene functions, predicting protein structures, and exploring evolutionary relationships across species.

Artificial Intelligence (AI) refers to the simulation of human intelligence in machines, enabling them to perform tasks such as reasoning, learning, problem-solving, and decision-making. AI encompasses various techniques, including ML, deep learning, and natural language processing (NLP), to analyze complex data and automate decision-making. In GA, AI is used to enhance the efficiency of gene prediction, functional annotation, and data integration by processing large-scale biological datasets with minimal human intervention.

Machine Learning (ML) is a subset of AI that enables computers to learn patterns from data and make predictions or decisions without being explicitly programmed. In GA, ML algorithms are used to classify genes, predict functional elements, and enhance annotation accuracy by analyzing large-scale genomic datasets. ML approaches include supervised, unsupervised, and reinforcement learning, leveraging statistical models and neural networks to improve biological data interpretation.

Manual curation, also known as manual annotation, refers to the process in which human experts review, refine, and validate genome annotations to ensure accuracy and biological relevance. This process involves analyzing computationally generated annotations, resolving ambiguities, and incorporating insights from experimental data and scientific literature.

Human–AI collaboration (HAIC) refers to the dynamic interaction between humans and AI systems, where both work together toward overall goals by leveraging their complementary strengths. Unlike automation, where AI operates independently, or augmentation, where AI enhances human capabilities, HAIC involves a continuous exchange of information, decision-making, and adaptation over time.

Knowledge graphs (KGs) are structured representations of relationships between biological entities, such as variants, genes, proteins, pathways, phenotypes, and diseases. They encode known interactions and associations in a graph format, where nodes represent entities and edges denote relationships. KGs facilitate data integration, reasoning, and discovery in genomics by linking heterogeneous biological information sources.

Large language models (LLMs) are AI models trained on massive datasets of text and code. They can generate human-quality text, translate languages, follow user instructions for task procedures [99, 100], use external tools [101], and answer user questions based on specific contexts [102]. A common architectural foundation for LLMs is the Transformer [103], which enables efficient modeling of long-range dependencies in sequences through self-attention mechanisms. Variations of this architecture include encoder-only models (e.g. BERT [61]), decoder-only models (e.g. GPT [62]), and encoder–decoder hybrids (e.g. T5 [104]). These architectures may be bidirectional, capturing context from both preceding and following tokens (as in BERT), or unidirectional, processing text left-to-right to generate coherent outputs (as in GPT models).

AI agent is an autonomous or semi-autonomous entity within a multi-agent system that performs specific tasks, interacts with other agents, and operates based on predefined rules, learned behaviors, or external inputs. Agents may specialize in different roles, such as task management, data retrieval, reasoning, or quality assessment, and they communicate within structured frameworks to enhance human–AI collaboration.

Key Points

• While genome annotation is complex and challenging, heavy reliance on automated methods can introduce errors.

• Manual curation is necessary for accurate annotations but requires significant time and effort.

• Our novel contribution is HAICoGA, the first conceptual framework for human–AI collaborative genome annotation.

• We further present a example of HAICoGA framework and future research directions in realize this framework.

Competing interests

The authors declare no competing interests.

Funding

None declared.

Data availability

All data presented in tables and figures were compiled from publicly available sources, which are cited in the manuscript and supplementary materials.