Applications of artificial intelligence for real-world evidence generation: a protocol for a living scoping review

Abstract

Introduction

Artificial intelligence (AI) is rapidly evolving, offering an expanding suite of capabilities that go beyond the traditional focus on prediction and classification. Generative AI (GenAI) and agentic AI could create transformative practices to support real-world evidence (RWE) generation for health research by streamlining studies, accelerating insights and improving decision-making. However, there is no published overview available describing the range of applications in RWE generation. This review aims to describe where and how genAI and agentic AI are applied across the domains of healthcare research tasks for RWE generation. Additionally, to map applications by tasks and methods across the product lifecycle continuum, and to identify emerging gaps and opportunities.

Methods and analysis

This Living Scoping Review (LSR) will include studies reporting an application and/or evaluation of genAI or agentic AI applied to one or more RWE generation research tasks. Searches will be conducted in Embase, MEDLINE and additional sources (eg, grey literature). Citations will be independently screened by two human senior reviewers for a substantive training dataset and a commercially available screening algorithm (Robot Screener) will complete screening with a human reviewer. The LSR will include reports of studies (primary or reviews) describing and/or evaluating the application of any genAI model for RWE generation in healthcare, in English, published from 1 January 2025 to the date of search. Data will be extracted from all studies included in the LSR by one independent senior reviewer using a piloted template, with 10% quality check by a second senior reviewer. Descriptive statistics will be used to summarise the applications of genAI per RWE research task, and the results of genAI evaluations. Thematic analysis will be used to describe genAI application patterns, trends, gaps and opportunities. The LSR protocol and reports will be updated annually, and findings will be published on a publicly available website (eg, ISPE—the International Society for Pharmacoepidemiology).

Ethics and dissemination

Ethical approval is not required due to use of previously published data. Planned dissemination includes peer-reviewed publication, presentation and short summaries.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• This review will be conducted following a living structured scoping approach, with planned periodic analyses, establishing a robust living evidence ecosystem to critically examine generative artificial intelligence (AI) applications to generate real-world evidence in healthcare research.

• The results of the search and the study selection process will be reported in full following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews.

• The protocol has had input from a multidisciplinary team of experts in healthcare research and generative AI.

• Only English reports will be included, meaning some other relevant reports published in other languages may not be included.

Introduction

Artificial intelligence (AI), including machine learning (ML), has increasingly important and varying applications in healthcare. Some applications are well established, such as clinical decision support systems,¹ natural language processing of records or image processing, while other applications are more recent, such as analysis of patient-generated data, support medical notes taking during patient visits and pharmacovigilance signal detection.²,⁴

Recently, generative AI (genAI), a type of AI that can create new content instead of just analysing or sorting existing data, has emerged as a transformative tool.⁵ At its core, genAI leverages advanced ML techniques to identify patterns, styles and structure from large amounts of existing data, then uses this knowledge to generate new content in the form of text (eg, generative pretrained transformer), images (eg, DALL·E), audio (eg, Suno), video (eg, Runway), code (eg, CodeWhisperer) and other digital content.⁶,¹¹ Large language models (LLMs), a type of genAI technology based on next-token prediction over large vocabularies, are widely used for natural language text processing and generation. They promise enhanced productivity and problem-solving benefits across various industries,^{12 13} particularly within healthcare research.¹⁴ LLMs are trained on a very large corpora of data using self-supervised learning, allowing them to learn patterns and structures directly from raw text without the need for explicit human-labelled data, and to generate new content in response to a specific (usually text) prompt.¹⁵

Although not yet as prevalent as in clinical medicine, AI pioneers have proposed the use of genAI in healthcare research for the generation of real-world evidence (RWE), the clinical evidence on the usage, risks and benefits of medical products by analysis of real-world data (RWD). RWE generation involves several complex tasks related to research study design and conduct that could be supported by genAI, such as research participant recruitment and retention, collection and analysis of patient-level primary data (direct to participant studies or site-based studies) as well as secondary data from electronic health records, administrative claims data, patient registries and other real-world datasets.^{7 16} Secondary research such as evidence syntheses (ie, literature reviews and meta-analyses) of primary RWE generation studies could also benefit from genAI in tasks such as: search strategy development, search output screening, data extraction from included studies and reporting. In this context, genAI could be used to accelerate infectious disease epidemic modelling,¹⁷ enhance drug development, improve drug evaluation¹⁸,²⁰ and streamline evidence synthesis.²¹,²⁵ LLMs, in particular, have shown promise in automating the extraction and analysis of large secondary datasets, such as unstructured patient medical data, and in drug safety surveillance data.^{7 22}

While genAI has great potential in enhancing both RWD and RWE generation, it also presents potential risks and limitations that require careful oversight and validation. The challenge of genAI lies in its ability to generate coherent, accurate and contextually appropriate content requiring careful oversight and validation (eg, hallucinations that can introduce inaccuracies in genAI applications).^{3 26} Position statements on the use of genAI in RWE have already been published from regulatory and health technology assessment (HTA) agencies.¹⁸²⁷,²⁹ However, despite the pace at which genAI in RWE research is proliferating, and the anticipation and uncertainties around genAI implementation in research, there exists no overview of how genAI is currently being used in RWE generation, or evaluation of model results regarding accuracy and efficiency.

A living scoping review (LSR) could provide a contemporary landscape analysis of genAI in healthcare research for RWE generation, addressing this evidence gap.³⁰ Living reviews are designed to be updated at repeated intervals and are well suited for rapidly evolving and novel subjects such as AI, where static reviews may quickly become obsolete.^{31 32} A LSR approach can help stay abreast of the state of the art and guide future developments in a dynamic field, including the development of research governance and design/conduct/reporting guidance documents for best practices.

Objectives

This LSR aims to identify where and describe (map) how genAI and agentic* AI is being applied across the spectrum of RWE generation research tasks, including hypothesis formulation and refinement, study planning and design, feasibility assessment, study conduct including data identification/collection/management and data analyses, and research reporting.

*This first iteration (V.1.0) of our LSR primarily focuses on genAI, reflecting its more established presence and broader base of existing knowledge. However, we have also incorporated agentic AI terms into our search strategy, given the growing relevance and evolving applications of agentic AI in both current and anticipated real-world research. As publications related to agentic AI are expected to increase over time, we plan to expand the inclusion of this literature in the second round of the LSR (V.2.0 of the LSR Protocol). Agentic AI refers to augmented LLM systems capable of autonomous decision-making and multistep task execution. Multi-agent systems (Agentic AI) take this further by deploying teams or groups of specialised AI agents that collaborate to address complex challenges. These approaches are theorised to deliver greater adaptability and effectiveness in generating complex RWE.

Specifically, the LSR aims to answer the following questions based on available literature:

• Where have genAI and agentic AI applications been used for specific RWE generation research tasks across the RWD to RWE continuum?

  • What are the common categories of use (how are they being used) and their applications?

  • What are the common model architectures used, how are they trained and what types of data are these tools trained on?

  • What are the key limitations and risks associated with these applications?

• What is the performance in RWE generation research tasks, regarding the model’s credibility and accuracy, and how has this been evaluated in the literature?

• What are established patterns and emerging trends associated with applications in RWE generation?

• Where are the gaps and opportunities that exist for applications in RWE generation lifecycle across the continuum of research tasks?

Methods and analysis

The LSR will be conducted and reported in accordance with the Joanna Briggs Institute methodology for scoping reviews as well as the Cochrane guidance for living reviews.^{31 33} The protocol will be registered on Open Science Framework (OSF).³⁴ This will facilitate protocol updates in future iterations of the LSR. The first iteration (V.1.0) of the protocol will be published in a peer-reviewed scientific journal.

Inclusion and exclusion criteria

In the first iteration of the LSR, reports fulfilling the following criteria will be included:

• Report the application and/or evaluation of agentic and/or genAI (genAI may be a component within an agentic AI system, providing content or analysis, but agentic AI coordinates and orchestrates broader activities beyond content generation), where genAI is defined as a class of AI models capable of generating new content by learning patterns from existing datasets, including LLMs and diffusion models, agentic AI is defined as AI systems endowed with higher levels of autonomy and decision-making capabilities that can set goals, make choices and execute multi-step tasks with minimal human intervention. Agentic AI is characterised by agency, that is, the capacity to act independently, plan, learn from feedback and adapt to changing environments, and

• The agentic AI and/or genAI is applied to one or more RWE generation research tasks, defined as study conceptualisation, design, conduct and reporting, of studies using routinely collected RWD and/or the evidence synthesis of such studies.

Studies that evaluate the model with any quality or performance focus (eg, model accuracy, credibility or efficiency) using any quantitative or qualitative design (including case studies) will be included.

Regarding publication types, this LSR will include journal articles, published conference abstracts and grey (ie, non-peer reviewed) literature (eg, preprints). The LSR will include original research studies. Any secondary research (eg, literature reviews, guidelines) identified in the search will be retained, to check the reference lists for eligible original research studies.

Reports will be excluded from the LSR, if they:

• Report the application of AI and ML applications that do not qualify as genAI (eg, discriminative model-based classifiers) nor agentic AI,

• report the application of genAI or agentic AI for purposes other than RWE generation (eg, applications in clinical practice or in clinical trials),

• are in languages other than English,

• were published before 1 January 2025. This limit has been selected to ensure the feasibility of the first iteration of the LSR, and to minimise the number of now obsolete genAI applications in the first iteration of the LSR.

In planned updates of the LSR, the eligibility criteria may be modified as needed (eg, the publication date limit will be updated, to include only new reports published between LSR iterations). All amendments will be recorded in the living protocol document on OSF.

Literature search

A multifaceted approach will be taken to identify literature eligible for inclusion in the LSR, including electronic databases, grey literature and backward snowball searches (sources illustrated in figure 1). The preliminary search strategy for Embase is presented in online supplemental appendix I. As no validated search filters exist for genAI and for RWE, the search terms were selected based on the search filter for genAI developed by the University of Alberta,¹¹ analysis of relevant studies through the MeSH Analysis Grid Generator¹² and a recently published taxonomy.³⁵

Figure 1. Overview of the living scoping review search strategy. *In the first iteration of the LSR, Google searches will additionally aim to identify sources where relevant information may be published (e.g.; academic blogs on genAI and/or agentic AI in healthcare research). Such sources will be incorporated in the search strategy of LSR updates. The following information will be recorded for each search: date, search terms, stopping criteria (eg, first two pages of results). genAI, generative artificial intelligence; LSR, Living Scoping Review.

Report selection

The search results will be exported from OVID SP and uploaded to the Nested Knowledge (NK) platform (https://nested-knowledge.com/) for further deduplication and screening. NK uses AI to learn from screening decisions within specific projects, generating inclusion probabilities for each record that requires screening, based on user configurations. These classifiers automate the screening process by replacing human reviewers with the Robot Screener, which makes inclusion and exclusion decisions based on the generated probabilities.^{35 36}

First, two human reviewers will independently screen 5%–10% of the title/abstract records. Any conflicts will be resolved by consensus and, if needed, discussion with a third reviewer. The data will be used as the training set for the Robot Screener. All remaining titles and abstracts will be screened by one experienced reviewer, and Robot Screener will be used as the second reviewer to quality-check inclusion and exclusion decisions. Conflicts between human reviewers and Robot Screener decisions will be resolved by a second human reviewer. The proposed training set size (5%–10% of all title/abstracts) was selected based on published studies using NK.^{37 38} The full texts of selected citations will be assessed in detail against the inclusion criteria by one experienced reviewer, with 20% quality check by a second reviewer. Reasons for full text exclusion will be recorded. The results of the search and the study selection process will be reported in full in the report and presented in a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.³⁹

Data extraction and analysis

The LSR aims to identify a broad range of AI tools that have been developed and/or used to support various RWE research tasks. The primary aim of the data extraction is to provide a detailed description of the application of the genAI and agentic AI tools for RWE generation. To achieve this, the LSR will include both quantitative and qualitative data extraction and analysis steps.

The draft data extraction tool will be modified and revised as necessary during the process of data extraction. Extraction will be completed by one experienced reviewer, with 10% QC by a second senior reviewer and resolution of any disagreements through discussion or resolution by a third expert reviewer. If appropriate, authors of papers or genAI manufacturers/developers will be contacted to request missing or additional data, where required.

Quantitative data extraction

Data will be extracted from included reports by one independent reviewer and quality-checked by a second independent reviewer, on Nested Knowledge using a custom data extraction template. The data extracted will include:

• General characteristics of the included report (ie, title, date of publication, journal, sponsor etc)

• Characteristics of the genAI or agentic AI tool(s) (ie, name, developer, model architecture, training set)

  • Where applicable, characteristics of and methods for the genAI or agentic AI tool (ie, usability, feasibility, accuracy, efficacy/effectiveness research, with further information on the tool development, evaluation and tool testing methodology)

• The RWE generation use case described in the report (ie, the specific research task(s) for RWE generation for which the tool was applied, the broader RWE generation design/method, the point(s) in the RWE generation lifecycle)

  • Where applicable, clinical/medical characteristics (the therapeutic domain/area, intervention(s)/comparator(s))

• Narrative summary of key quantitative findings relevant to the review objectives.

Qualitative data extraction

To identify:

• current patterns of use and emerging trends for the application of tools for the spectrum of RWE generation tasks (eg, hypothesis formulation and refinement, study planning and design, feasibility assessment, study conduct including data identification/collection/management and data analyses, and research reporting),

• as well as gaps and opportunities for potential tools and uses for RWE generation across the product lifecycle, we will conduct a thematic analysis and synthesis of the included reports.³⁹

Each included full-text report will undergo coding and analysis.⁴⁰ First, one independent experienced review author will inductively code a set of 10 reports line-by-line (specifically, the Results and Discussion sections covering the authors’ interpretation of their data). Second, the reviewer will independently organise the open codes based on similarities and differences between codes into structured descriptive subthemes and themes that will map models and their application across the spectrum of RWE generation tasks and broader types of study designs respectively. Third, a second senior reviewer will join to discuss and reach consensus on the codes and descriptive subthemes/themes.

The second reviewer will also engage in further interpretative discussion focused on the research objective and generate analytical subthemes and themes to map gaps and opportunities for potential uses of RWE across the product lifecycle.

Subsequently, the first reviewer will independently code all remaining reports, adding new excerpts to the pre-existing codes and themes in the codebook as well as creating new codes and themes as appropriate.

Following extraction, data will be tabulated into a thematic tree and figures. Based on the gap analysis, we will present recommendations and opportunities for RWE researchers, RWE research end-users and genAI developers.

Annual LSR update process

This LSR will be updated annually for the next 5 years as a minimum. Each update will comprise the following steps:

1. Publication of the updated protocol on OSF, reporting any amendments to the previous version of the LSR protocol.

2. Re-run of the database searches via OVID and search results export. The search results will be manually imported to NK and automatically deduplicated.

3. After deduplication, all new titles and abstracts will be screened on NK. This step will be semi-automated with reviewer oversight. Specifically, the NK classifier, trained on screening decisions of previous iterations of the review, will be used to automatically include and exclude records. To ensure no records are incorrectly excluded, an experienced reviewer will independently screen 10% of all records. If the process identifies false negatives, an additional 10% will be screened, until no false negatives are identified.

4. All records selected for inclusion will be manually checked by an experienced reviewer before data extraction.

5. Quantitative and qualitative data will be manually extracted from newly included studies, and the new evidence will be manually incorporated in the LSR report.

6. The updated results and report, as well as any protocol amendments, will be uploaded to OSF and disseminated via a publicly available website.

Patient and public involvement

None.

Ethics and dissemination

Ethical approval is not required to conduct this scoping review, which will use only previously published data. Our dissemination strategy includes peer-reviewed publication, presentation at conferences and short summaries in lay language on professional social media.

Discussion

This study protocol outlines a living and structured scoping approach to critically examine and explore genAI and agentic AI applications to generate RWE in healthcare research. Through periodic analysis of the literature, this LSR will deliver rapid and rigorous insights into the role in assisting RWE generation, across the spectrum of real-world study designs and methodologies that are used for decision-making across the product lifecycle in healthcare research. The review will identify key patterns and trends (established and emerging), as well as potential risks/limitations that are quality/performance related, serving as a foundation for developing recommendations on benchmarking standards, regulation and governance for genAI and agentic AI use in RWE generation, and possibly shape future directions where RWE research could benefit. The review will also uncover gaps, highlighting unmet need and opportunities for genAI and agentic AI development and improvement. The results of the search and the study selection process will be reported in full following the PRISMA extension for scoping reviews. However, only English-language reports will be included, meaning some other relevant reports published in other languages may not be included. Further, the protocol has had input from a multidisciplinary team of experts in healthcare research aiming to inform the development of guidance to support transparent and complete reporting of the use of genAI and agentic AI in RWE generation in healthcare research. Ultimately, establishing a robust living evidence ecosystem of genAI and agentic AI applications for RWE generation in healthcare research will contribute to generating more timely and impactful RWD and RWE for drug development and informing regulatory, HTA, research, clinical and patient decision-making.

The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, the FDA/HHS, or the US Government.