Abstract
Background
Early-phase oncology trials involve complex protocols and extensive documents, making timely resolution of study queries challenging. We developed the Study Document Assistant (SDA), a retrieval-augmented generation (RAG) system that integrates semantic search with large language models to generate context-specific answers from clinical trial documentation.
Materials and methods
SDA processes study documents and delivers responses through a secure, conversational interface. A two-arm experiment was conducted in which sponsor clinical study team members answered protocol-related queries either with or without SDA. Questions were stratified by difficulty and assessed by blinded subject matter experts (SMEs) and artificial intelligence (AI) models. The primary endpoint was mean response time. Main secondary endpoints included response time as a time-to-event outcome and overall accuracy. SDA user satisfaction was also assessed. Data were analyzed using Welch t-test for the primary endpoint, Kaplan–Meier method and log-rank test for time-to-event analysis, Mann–Whitney U test for accuracy, and descriptive statistics for satisfaction.
Results
In the experiment (N = 10 per arm), SDA users achieved a mean response time of 14.9 [95% confidence interval (CI) 9.3-20.6] versus 26.5 (95% CI 16.8-36.1) minutes in controls, corresponding to a 43.8% reduction (95% CI 11.9% to 64%, P = 0.0347). A similar reduction for median time to response was observed. Accuracy, measured as the median SME score (scale 1-4), was ∼3 (“Mostly Correct”) for both arms, with no statistically significant difference. Evaluation by AI models showed similar results.
Conclusions
Our findings demonstrate that SDA accelerates query resolution without compromising answer quality, highlighting the potential of RAG-based digital assistants to streamline clinical trial operations and enhance research efficiency.
Key words: clinical trials, digital health, large language model, oncology, retrieval-augmented generation, study document assistant
Highlights
-
•
We developed SDA, a cloud-native RAG system using LLMs for clinical trial queries.
-
•
SDA reduced mean response time by 43.8% versus manual review in first-time users.
-
•
SDA maintained response quality, as confirmed by independent SMEs and AI models.
-
•
SDA delivered high user satisfaction and smooth deployment.
Introduction
Early oncology clinical trials are critical in the advancement of cancer treatments, primarily focusing on assessing the safety, efficacy, and pharmacokinetic/pharmacodynamic properties of novel therapies.1 The design complexity of these trials is increasing, with seamless, modular protocols that are hard to navigate and interrogate.2 Moreover, associated documents, including investigator brochure, informed consent, toxicity management guidelines, and statistical analysis plan, among others, add complexity to query answering and document consultation, potentially slowing down trial execution.2 A significant challenge arises from the frequent need for investigators and study site personnel to seek clarity on issues such as patient eligibility criteria, treatment schedules, protocol amendments, informed consents, and adverse event management. Any query related to the clinical study may require a significant amount of time to be addressed given the complexity, quantity, and length of clinical trial documentation.3 Providing prompt and precise answers to these queries is vital to maintain protocol adherence, ensure patient safety, and expedite recruitment timelines.3
Traditionally, resolving these queries has been labor-intensive, relying on manual searches through extensive trial documents or expert consultations, which can result in delays and operational inefficiencies. Factors such as the growing volume of trial-related data, stringent regulatory requirements, and the need for real-time decision making make it difficult for the sponsor personnel to efficiently identify, interpret, and act upon critical information.4 These inefficiencies can contribute to operational bottlenecks, ultimately delaying the progress of novel treatments for cancer patients.5
Recent advancements in artificial intelligence (AI) and large language models (LLMs) based on the transformer architecture have shown promise in tackling these challenges.6 Specifically, when integrated with a retrieval-augmented generation (RAG) framework, LLMs have the potential to both retrieve relevant data from structured sources and generate precise, context-aware responses to complex queries.7 This hybrid approach is particularly advantageous for knowledge-intensive tasks, such as resolving queries in oncology trials.
Starting from the unmet need of reducing the operational workload on early oncology clinical and operational teams while minimizing the cognitive burden, we designed and tested an LLM-enhanced RAG application called the Study Document Assistant (SDA) to efficiently address queries from health care personnel during early-phase oncology trials. Specifically, we aimed to streamline the query resolution process and reduce the time needed to query resolution while preserving the accuracy of the answers provided.
In this article, we describe the development and piloting evaluation of SDA and preliminary evidence of its utility in terms of time saving and accuracy in answering real queries from users who are unfamiliar with the specific study and have not previously used the tool.
Materials and methods
The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis-Large Language Models (TRIPOD-LLM) reporting guideline for studies using LLMs, as well as the specific recommendations for reporting digital health and AI research in oncology provided by the ESMO Guidance for Reporting Oncology real-World Evidence (ESMO-GROW), were followed for the present work.8,9
SDA architecture and core components
SDA is implemented as a cloud-native RAG system on Amazon Web Services, specifically engineered to handle the complex information retrieval needs of clinical trial management. Study documents are stored in Amazon S3 with automated preprocessing pipelines that extract, chunk, and vectorize textual content for optimal retrieval performance. The architecture consists of a containerized microservices deployment using Docker technology, with separate containers optimized for document processing, vector search, and response generation (Supplementary Figure S1, available at https://doi.org/10.1016/j.esmorw.2026.100726). Model orchestration is provided via Amazon Bedrock with task-optimized selection across general and biomedical foundation models. Detailed model roster, routing logic, and prompt schemas are described in Supplementary Methods, available at https://doi.org/10.1016/j.esmorw.2026.100726. SDA operates as a protocol-agnostic, zero-shot RAG service and incorporates query and response guardrails to mitigate unsafe or weakly grounded outputs; full specifications are described in Supplementary Methods, available at https://doi.org/10.1016/j.esmorw.2026.100726.
User authentication and role-based access control are managed through Amazon Cognito, ensuring appropriate access to sensitive clinical documentation based on user credentials and study assignments.
Evaluation framework
We conducted a two-arm, controlled experiment to demonstrate the tool’s capability to streamline routine information retrieval tasks. Participants were allocated to one of two groups: Arm A, in which clinical study team members responded to queries using the clinical question-answering tool (SDA), and Arm B, in which clinical study team members responded to queries without the use of SDA. Participants in Arm B accessed study documents in PDF format via Adobe Reader and used standard keyword search functions. Participant selection is detailed in Supplementary Methods, available at https://doi.org/10.1016/j.esmorw.2026.100726. Of note, participants were stratified by role, and all were deliberately protocol-naïve to establish a standardized baseline. This design was chosen to avoid introducing bias arising from heterogeneous levels of protocol knowledge across participants, which could confound evaluation of the tool’s effectiveness.
The primary endpoint of this study was the reduction in mean time required to answer the three assigned questions. The secondary endpoints were the overall accuracy of participant responses and the response time treated as a time-to-event outcome. User-reported satisfaction in the SDA arm was an exploratory endpoint.
The study was conducted across sponsor clinical study team members encompassing various roles involved in real-life clinical queries answering. The distribution of the different functions was the same in both arms. Participants completed three clinical queries in a controlled environment during an online session and rated their satisfaction once per session.
Participants in both groups were challenged with three real questions received from study sites of increasing complexity and related to a pre-existing study protocol (Supplementary Material S1, available at https://doi.org/10.1016/j.esmorw.2026.100726). This study was designed to evaluate the performance of SDA in supporting the resolution of protocol clarification queries that relied exclusively on trial documentation. All test questions were selected such that accurate answers could be provided using the study documents, without requiring access to patient-specific data from electronic health records. The criteria employed to classify questions as easy, intermediate, or difficult were based on an external assessment and categorization conducted by expert members involved with a specific clinical protocol (see Supplementary Methods, available at https://doi.org/10.1016/j.esmorw.2026.100726).
Accuracy evaluation
The answers in both arms were analyzed to evaluate the secondary endpoint of accuracy, following standard approaches.10 Specifically, answers were evaluated through two different approaches: a human-based evaluation performed by two independent subject matter experts (SMEs) blinded to experiment arm and an answers evaluation by LLMs [ChatGPT-4.1 (OpenAI, San Francisco, CA) and Claude 4.0 Sonnet (Anthropic, San Francisco, CA)] versus a human-redacted, correct reference version of the answers (ground truth).
Retrieval effectiveness, answer grounding, and hallucination risk analysis
We evaluated SDA upstream of human verification using three complementary scores that capture retrieval, grounding, and hallucination characteristics of responses. Retrieval effectiveness (ground-truth coverage): the proportion of information in the reference answer that is present within the retrieved context (range 0-1; higher is better). Answer grounding: the proportion of content in the model’s answer that appears in the retrieved context (range 0-1; higher indicates stronger evidence support). Hallucination risk: a sentence-level measure that flags an answer sentence as potentially hallucinated when insufficient content from that sentence appears in the retrieved context. We assessed multiple SDA responses per query alongside their retrieved contexts and ground-truth references. Detailed computation procedures (tokenization, overlap operations), operational thresholds used for sentence-level decisions, and software versions are provided in Supplementary Methods, available at https://doi.org/10.1016/j.esmorw.2026.100726.
Evaluation criteria, metrics, and data collection
The following baseline data regarding the experiment participant as well as endpoints data were collected: study member team role, time to response, text of the responses to the questions, and tool satisfaction surveys for subjects in the SDA arm. Time-to-response data were captured in real time using Microsoft Forms, ensuring precision in the measurement of this key endpoint. Response quality ratings relied on accuracy and were assessed after the experiment both by independent, arm-blinded SMEs to ensure unbiased evaluation and by LLM tools (ChatGPT-4.1, Claude 4.0 Sonnet) using a Likert scale-based survey (Supplementary Material S2, available at https://doi.org/10.1016/j.esmorw.2026.100726). For accuracy evaluation by AI using LLMs, a reference correct answer was provided. The prompt for the evaluation is available in Supplementary Material S3, available at https://doi.org/10.1016/j.esmorw.2026.100726. To assess participant satisfaction and perceptions regarding the use of the clinical question-answering tool, the study utilized a Likert scale-based survey that measured overall satisfaction (Supplementary Material S4, available at https://doi.org/10.1016/j.esmorw.2026.100726).
Statistical analyses
Sample size determination and primary endpoint analysis
Benchmarking was performed to establish baseline response times without using AI support and to inform statistical assumptions for the two-arm controlled experiment (see Supplementary Methods, available at https://doi.org/10.1016/j.esmorw.2026.100726). The benchmark estimated a control-arm mean response time of 23.7 minutes [standard deviation (SD) 8.3 minutes]. Assuming a two-sided type I error rate of 5% and 80% power, a sample size of 10 participants per arm was calculated to detect a 40% reduction in mean response time in the experimental arm versus the control arm. The primary comparison of mean time to response employed a two-sample Welch t-test (t-test assuming unequal variances). Distributional assumptions were assessed using histograms and Shapiro–Wilk test. We report 95% confidence intervals (CIs) for arm-specific means (t-based with n-1 degrees of freedom) and for the between-arm mean difference using the Welch–Satterthwaite approximation. In addition, we quantified the relative effect as the percent reduction in mean response time and obtained a 95% CI by nonparametric bootstrap resampling at the participant level with 10 000 replicates (percentile method). Time-to-response distributions were additionally summarized using Kaplan–Meier methods, with between-arm comparisons assessed by the log-rank test. Analyses were performed in R (version 4.3.2).
Secondary endpoints and exploratory endpoints analyses
Secondary endpoints were explored to provide complementary evidence of the tool’s impact. These endpoints were not used to power the study and were not benchmarked. Accuracy was evaluated independently by two SMEs using a 1 to 4 Likert scale (1-Incorrect, 2-Partially Correct, 3-Mostly Correct, and 4-Completely Correct) for each query (Supplementary Material S2, available at https://doi.org/10.1016/j.esmorw.2026.100726). For each assessor, a composite accuracy score was defined as the average of the three per-query ratings. In parallel, accuracy was also scored by two LLMs (ChatGPT-4.1 and Claude 4.0 Sonnet) across five independent rounds each, and per-model composite accuracy scores were computed analogously. Between-arm differences in the distribution of the composite accuracy score were evaluated using the nonparametric Mann–Whitney U test. Satisfaction was captured once per session on a 1 to 5 Likert scale and summarized using descriptive statistics. No formal multiplicity adjustments were prespecified for secondary endpoints. Analyses were performed using R (version 4.3.2). The statistical significance threshold was set at a two-sided P < 0.05. For retrieval effectiveness, grounding quality and hallucination risk, means, SDs, and ranges for each metric across all answers were calculated. Analyses were performed using Python (version 3.10).
Results
SDA development
SDA is a RAG application designed to provide real-time support for clinical questions arising during any phase of clinical trials. SDA maintains a comprehensive repository of clinical study documents (Supplementary Table S1, available at https://doi.org/10.1016/j.esmorw.2026.100726) including protocols, investigator brochures, case report forms, regulatory correspondence, safety updates, and amendment histories.
The core RAG functionality combines semantic search capabilities with LLM reasoning to deliver rapid and contextually accurate responses. When users submit clinical questions, the system performs multistage retrieval and answering (Figure 1). Briefly, in the query processing stage, natural language queries are analyzed and expanded with clinical terminology and synonyms. During document retrieval, semantic search across vectorized document embeddings identifies relevant protocol sections and supporting materials, whereas in context assembly, retrieved passages are ranked and assembled into coherent context windows. Lastly, during response generation, LLMs generate responses grounded in the retrieved clinical documentation. SDA presents answers and their provenance within a single, integrated workspace to support claim-level verification at the point of use. The synthesized answer is rendered in a panel in prose form, with discreet inline citation badges that map each sentence or clause to one or more source passages. A second panel opens a synchronized document viewer at the exact locations used for generation; the retrieved passages are highlighted within their native documents and accompanied by metadata, including document title and version, section or page markers, and a persistent link to the source artifact. This design makes the linkage from answer to source explicit, navigable, and auditable, operationalizing the transparency and traceability expectations.
Figure 1.
Overview of the SDA question-answering workflow. The pipeline consists of four sequential stages. In the query processing stage, natural language queries are analyzed and expanded with clinical terminology and synonyms. During document retrieval, a semantic search across vectorized document embeddings identifies relevant protocol sections and supporting materials. In the context assembly stage, retrieved passages are ranked and assembled into coherent context windows. Finally, during response generation, LLMs generate answers grounded in the retrieved clinical documentation.
LLM, large language model; SDA, Study Document Assistant.
SDA evaluation
The experiment met its primary endpoint, with a significant reduction in mean response times for participants using SDA compared with the control group. These results specifically reflect performance on protocol-based queries, with all test questions resolved using study documents alone and no patient-specific data integration. Specifically, the mean response time in Arm A (SDA users) was 14.9 minutes (95% CI 9.3-20.6) compared with 26.5 minutes (95% CI 16.8-36.1) in Arm B (control), corresponding to a 43.8 % reduction (95% CI 1.0% to 22.1%, Welch t-test P = 0.0347) (Figure 2A). Distributions were approximately normal in both arms with no material skewness or influential outliers (Shapiro–Wilk: control W = 0.929, P = 0.438; SDA W = 0.939, P = 0.545) (Supplementary Figure S2, available at https://doi.org/10.1016/j.esmorw.2026.100726), supporting the use of a Welch two-sample t-test given unequal variances. A time-to-event analysis (treating the completion of the task of answering the three questions as the event) further supported faster response times with SDA (median time to response: 14.2 minutes for SDA users versus 25.5 for the control, log-rank P = 0.0263; Figure 2B).
Figure 2.
SDA tool significantly reduces clinical question response times. (A) Mean time to response was significantly shorter in Arm A (SDA users) at 14.9 minutes (95% CI 9.3-20.6) compared with 26.5 minutes (95% CI 16.8-36.1) in Arm B (control), representing a 43.8% reduction (95% CI 1.0% to 22.1%, Welch t-test P = 0.03467). Red diamonds show means; boxes show median and quartiles. (B) Median time to response was similarly reduced, with 14.2 minutes in Arm A versus 25.5 minutes in Arm B (P = 0.0263).
CI, confidence interval; SDA, Study Document Assistant.
Accuracy, measured as the average accuracy of all three questions, was preserved in the experiment using SDA. No statistically significant differences were observed between Arm A (SDA users) and Arm B (control) when the evaluation was performed by blinded human SMEs (assessor 1: median accuracy score = 3 for Arm A versus 3 for Arm B, P = 0.1888; assessor 2: mean accuracy = 2.97 for Arm A versus 3.3 for Arm B, P = 0.2596; Figure 3) with median accuracy score ∼3 (i.e. answers Mostly Correct) for both arm. Similar results about overall accuracy were observed when the evaluation was performed by two AI models (Claude 4.0 Sonnet and ChatGPT-4.1) in multiple rounds (Supplementary Tables S2 and S3, available at https://doi.org/10.1016/j.esmorw.2026.100726).
Figure 3.
Accuracy of responses was maintained with SDA. Overall accuracy, (measured as accuracy score, i.e. the mean Likert-scale score across three clinical questions), showed no statistically significant difference between Arm A (SDA users) and Arm B (control). Evaluations by human subject matter experts (SMEs) yielded median accuracy scores of 3 versus 3 (P = 0.1888) for assessor 1 and 2.97 versus 3.3 (P = 0.2596) for assessor 2, indicating that use of SDA preserved response accuracy, with median accuracy score ∼3 (i.e. answers ‘Mostly Correct’) for both arms. Medians shown as white circles.
SDA, Study Document Assistant.
Complementary evaluation of SDA using the same questions of the experiment (10 iterations per question) revealed promising performance in hallucination control and moderate performance in retrieval effectiveness. SDA achieved a mean hallucination rate of 1.9% (SD ±4.3%, range 0.000-0.125). Mean answer grounding was 59.5% (SD ±14.6%, range 0.364-0.840), indicating that approximately 60% of generated content was directly supported by retrieved evidence. Mean retrieval quality, measured as ground truth coverage, was 48.5% (SD ±7.0%, range 0.400-0.658), suggesting moderate success in identifying relevant information with opportunity for optimization.
Exploratory tool user satisfaction analysis in the experiment tasks for participants in the SDA arm showed that of 10 participants, 8 (80%) were satisfied or very satisfied by the SDA user experience (Supplementary Table S4, available at https://doi.org/10.1016/j.esmorw.2026.100726).
Overall, SDA achieved approximately 40%-45% faster responses compared with standard practice, with accuracy maintained and user satisfaction at high levels.
Discussion
AI is transforming oncology and the conduct of clinical trials across multiple dimensions. The expanding suite of advanced technologies enables automation and optimization of traditionally complex, resource-intensive, and time-consuming processes, thereby enhancing overall efficiency and accelerating study timelines.11
This study presents the development and evaluation of SDA, a RAG application tailored to address queries in early oncology clinical trials. SDA functions as a question-answering assistant rather than a general-purpose chatbot. Users pose natural-language queries; the system retrieves relevant passages from the uploaded study documents and synthesizes a grounded response with citations. In a two-arm, controlled experiment, we demonstrated that the use of this application significantly reduced response times while preserving the accuracy of responses compared with conventional methods, highlighting its potential in streamlining communication between health care professionals and study sponsor personnel. Of note, in both study arms, the mean accuracy of responses was 3 of 4 (i.e. Mostly Correct), with neither group achieving the highest score (i.e. Completely Correct). This ‘ceiling effect’ likely reflects the inherent complexity of the protocol used, the cognitive challenges of synthesizing information from multiple documents, and the deliberate use of protocol-naïve participants with limited training. Additionally, the stringent scoring rubric may have contributed, as the threshold for Completely Correct responses required exhaustive detail and the absence of any minor omission. These findings highlight that, even with advanced tools, answering complex protocol queries remains inherently challenging.
The complimentary analysis of SDA retrieval effectiveness, grounding quality, and hallucination risk indicates a prioritization of factual reliability while maintaining competitive performance in other dimensions. The promising hallucination control positions the system favorably for clinical applications, in which hallucinations are primary concerns.12,13 The moderate retrieval performance, while an area for improvement, represents a tractable optimization challenge amenable to established techniques such as hybrid search, re-ranking, and domain-specific embedding fine-tuning.14,15 These results are in line with a rapidly expanding body of literature underscoring the potential of AI-driven tools (especially LLMs and RAG frameworks) to improve efficiencies across complex clinical workflows and decision-making processes.4,7,16 The considerable reduction in time to response and the consistent maintenance of response quality confirms the utility of the RAG framework in enhancing both the efficiency and effectiveness of query resolution. This is supported by recent systematic reviews that show that existing LLM applications in health care consistently achieve high accuracy in knowledge retrieval and diagnostic support but often lack validation in real-world workflows.17 By demonstrating operational gains in a live clinical trial setting, our study directly addresses this critical gap and aligns with previous studies suggesting that integrating LLMs with retrieval mechanisms can facilitate precise, context-specific information retrieval in health care settings.7,18,19 SDA demonstrated its potential to facilitate real-time decision making in clinical settings, particularly when rapid and accurate information retrieval is crucial. Its scalability and adaptability make it a viable solution for broader clinical operations across various therapeutic domains.
Recent evaluations of advanced LLM-based tools have established the capability of AI systems not only to match but sometimes surpass physicians in diagnostic accuracy and multidimensional clinical competencies during simulated patient–physician interactions.16,18 Our results complement these findings by demonstrating how similar architectures, when deployed for operational support in research, can transform the efficiency of sponsor–investigator communication, crucial for successful clinical trial execution. Additionally, the broader role of AI in drug development and clinical research highlights the scalability and adaptability of these systems as foundational for generalized clinical operations, therapy discovery, and data-driven decision making.16,19, 20, 21, 22 The application flexibility of SDA aligns with these advances, suggesting viability as a plug-in augmentation across diverse clinical domains. However, integrating such AI-driven applications into existing systems requires holistic planning, thoughtful workflow adjustments, and robust user training to maximize benefits and user acceptance, as emphasized by recent reporting guidelines and practical studies.8,23,24
Despite the promising findings, some limitations must be acknowledged. Firstly, the study was conducted within a specific context of early-phase oncology clinical trials and using a single clinical study, which may limit the generalizability of the results to other therapeutic domains. Additionally, participants were not given prior training on SDA, which might have influenced the learning curve and effectiveness of using the RAG application. This challenge is echoed in recent literature indicating that user training and experience remain underexplored factors in real-world AI impact.17 Notably, our study evaluated SDA implementation in a protocol-naïve user scenario. Although this may mean that the relative efficiency gains observed could be reduced in a real-world setting (where personnel typically have accumulated protocol knowledge), it also highlights a potential strength of the RAG-based tool: personnel without prior specific or in-depth training on study documents were able to efficiently resolve queries. This suggests that the tool may facilitate onboarding or support for new team members who lack immediate protocol expertise. Another limitation is that the experiment exclusively involved study document-related and procedural queries, rather than site-level, patient-specific clinical queries that usually contribute to the operational bottleneck in clinical trial management. Future work should evaluate the performance and impact of SDA or similar tools in scenarios requiring real-time access to patient data. Moreover, we acknowledge that the complementary retrieval and hallucination analysis covered only three questions with repeated runs and that broader, multiprotocol evaluations are needed to confirm the generalizability of these metrics.
The study’s sample size, while adequately powered for the primary endpoints, may not fully capture long-term operational impacts. As seen in prior work, expanding the diversity of trial data and sharing implementation insights will be crucial for broader generalization and fairness of these applications.18,20,21 Building on these considerations, future research should explore the utility of SDA in a broader range of clinical settings and therapeutic areas. Looking ahead, subsequent work will focus on extending the SDA framework beyond study protocols to encompass additional domains of clinical operations, including safety reporting, site management, and regulatory documentation. This progression will represent a logical evolution toward a more comprehensive tool capable of supporting end-to-end trial activities. In parallel, emerging advances in agentic AI, systems that can reason, plan, and act autonomously within predefined parameters, may further enhance the capabilities of SDA applications.25 Integrating SDA architecture within an agentic framework could enable proactive query resolution, dynamic workflow orchestration, and continuous learning from user interactions.26
Conclusions
SDA, a newly developed RAG application, demonstrated promising potential for enhancing the efficiency of query resolution within early-phase oncology trials in a pilot, controlled setting. Although these results suggest opportunities for broader integration of advanced AI applications into clinical research and operations, the findings should be interpreted in light of the study’s limited sample size, specific operational context, and focus on early oncology. Further studies with larger, more diverse participant populations, across multiple clinical and geographic contexts, are warranted to extend its application scope, address remaining challenges, and ensure that advances in AI for health care remain robust, equitable, and truly transformative for all stakeholders.
Acknowledgements
The authors would like to thank all participants in the evaluation experiment.
Data and code availability
The datasets generated and/or analyzed during the present study are not publicly available due to confidentiality restrictions, but they may be made available from the corresponding author to qualified researchers on reasonable request. The underlying code for this study is not publicly available for proprietary reasons.
Funding
This study was funded by AstraZeneca and Evinova (no grant number).
Disclosure
DP, GF, AM, LK, and MM are full-time employees of AstraZeneca and may have stock ownership, options, or interests in the company. KB, AS, MK, and FM are employees of Evinova and may have stock ownership, options, or interests in the company. The authors declare no other conflicts of interest related to this publication.
Supplementary data
References
- 1.Cook N., Hansen A.R., Siu L.L., Abdul Razak A.R. Early phase clinical trials to identify optimal dosing and safety. Mol Oncol. 2015;9(5):997–1007. doi: 10.1016/j.molonc.2014.07.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Araujo D., Greystoke A., Bates S., et al. Oncology phase I trial design and conduct: time for a change – MDICT Guidelines 2022. Ann Oncol. 2023;34(1):48–60. doi: 10.1016/j.annonc.2022.09.158. [DOI] [PubMed] [Google Scholar]
- 3.Peralta G., Sánchez-Santiago B. Navigating the challenges of clinical trial professionals in the healthcare sector. Front Med (Lausanne) 2024;11 doi: 10.3389/fmed.2024.1400585. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Esteva A., Robicquet A., Ramsundar B., et al. A guide to deep learning in healthcare. Nat Med. 2019;25(1):24–29. doi: 10.1038/s41591-018-0316-z. [DOI] [PubMed] [Google Scholar]
- 5.Bieske L., Zinner M., Dahlhausen F., Truebel H. Critical path activities in clinical trial setup and conduct: how to avoid bottlenecks and accelerate clinical trials. Drug Discov Today. 2023;28(10) doi: 10.1016/j.drudis.2023.103733. [DOI] [PubMed] [Google Scholar]
- 6.Vaswani A., Shazeer N., Parmar N., et al. Attention is all you need [preprint] https://ui.adsabs.harvard.edu/abs/2017arXiv170603762V arXiv. 2017;arXiv:1706.03762. Submitted June 2017, last revised August 2, 2023. Available at.
- 7.Lewis P., Perez E., Piktus A., et al. Retrieval-augmented generation for knowledge-intensive NLP tasks [preprint] https://ui.adsabs.harvard.edu/abs/2020arXiv200511401L arXiv. 2020;arXiv:2005.11401. Submitted May 2020, last revised April 12, 2021. Available at.
- 8.Gallifant J., Afshar M., Ameen S., et al. The TRIPOD-LLM reporting guideline for studies using large language models. Nat Med. 2025;31(1):60–69. doi: 10.1038/s41591-024-03425-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Castelo-Branco L., Pellat A., Martins-Branco D., et al. ESMO Guidance for Reporting Oncology real-World evidence (GROW) ESMO Real World Data Digit Oncol. 2023;1 doi: 10.1016/j.esmorw.2023.10.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Desmond M., Ashktorab Z., Pan Q., Dugan C., Johnson J.M. March 18-21, 2024. EvaluLLM: LLM assisted evaluation of generative outputs. Presented at IUI ‘24 Companion: Companion Proceedings of the 29th International Conference on Intelligent User Interfaces. Greenville, SC, USA. [Google Scholar]
- 11.Chow R., Midroni J., Kaur J., et al. Use of artificial intelligence for cancer clinical trial enrollment: a systematic review and meta-analysis. J Natl Cancer Inst. 2023;115(4):365–374. doi: 10.1093/jnci/djad013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Jin Q., Yang Y., Chen Q., Lu Z. GeneGPT: augmenting large language models with domain tools for improved access to biomedical information. Bioinformatics. 2024;40 doi: 10.1093/bioinformatics/btae075. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Zakka C., Shad R., Chaurasia A., et al. Almanac — retrieval-augmented language models for clinical medicine. NEJM AI. 2024;1(2) doi: 10.1056/aioa2300068. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Gao L., Ma X., Lin J., Callan J. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Association for Computational Linguistics; Toronto, Canada: 2023. Precise zero-shot dense retrieval without relevance labels; pp. 1762–1777. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Gao Y., Xiong Y., Gao X., et al. Retrieval-augmented generation for large language models: a survey. https://ui.adsabs.harvard.edu/abs/2023arXiv231210997G arXiv. 2023;arXiv:2312.10997. Submitted May 2020, last revised March 27, 2024. Available at.
- 16.Teo Z.L., Thirunavukarasu A.J., Elangovan K., et al. Generative artificial intelligence in medicine. Nat Med. 2025;31(10):3270–3282. doi: 10.1038/s41591-025-03983-2. [DOI] [PubMed] [Google Scholar]
- 17.Bedi S., Liu Y., Orr-Ewing L., et al. Testing and evaluation of health care applications of large language models: a systematic review. JAMA. 2025;333(4):319–328. doi: 10.1001/jama.2024.21700. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Tu T., Schaekermann M., Palepu A., et al. Towards conversational diagnostic artificial intelligence. Nature. 2025;642(8067):442–450. doi: 10.1038/s41586-025-08866-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hung T.K.W., Mao J.J., Ho A.L., et al. Real-world evaluation of user engagement with an artificial intelligence-powered clinical trial application in oncology. J Am Med Inform Assoc. 2025;32(10):1562–1569. doi: 10.1093/jamia/ocaf129. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Zhang K., Yang X., Wang Y., et al. Artificial intelligence in drug development. Nat Med. 2025;31(1):45–59. doi: 10.1038/s41591-024-03434-4. [DOI] [PubMed] [Google Scholar]
- 21.Wang X., Zhao J., Marostica E., et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature. 2024;634(8035):970–978. doi: 10.1038/s41586-024-07894-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Xiang J., Wang X., Zhang X., et al. A vision-language foundation model for precision oncology. Nature. 2025;638(8051):769–778. doi: 10.1038/s41586-024-08378-w. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Wan P., Huang Z., Tang W., et al. Outpatient reception via collaboration between nurses and a large language model: a randomized controlled trial. Nat Med. 2024;30(10):2878–2885. doi: 10.1038/s41591-024-03148-7. [DOI] [PubMed] [Google Scholar]
- 24.Pascoe J.L., Lu L., Moore M.M., et al. Strategic considerations for selecting artificial intelligence solutions for institutional integration: a single-center experience. Mayo Clin Proc Digit Health. 2024;2(4):665–676. doi: 10.1016/j.mcpdig.2024.10.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Lakhan S.E. The agentic era: why biopharma must embrace artificial intelligence that acts, not just informs. Cureus. 2025;17(5) doi: 10.7759/cureus.83390. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Gorenshtein A., Omar M., Glicksberg B.S., Nadkarni G.N., Klang E. AI agents in clinical medicine: a systematic review. medRxiv. 2025 doi: 10.1101/2025.08.22.25334232. [preprint] 2025.08.22.25334232. Published August 26, 2025. Available at. [DOI] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The datasets generated and/or analyzed during the present study are not publicly available due to confidentiality restrictions, but they may be made available from the corresponding author to qualified researchers on reasonable request. The underlying code for this study is not publicly available for proprietary reasons.