RefAI: a GPT-powered retrieval-augmented generative tool for biomedical literature recommendation and summarization

Abstract

Objectives

Precise literature recommendation and summarization are crucial for biomedical professionals. While the latest iteration of generative pretrained transformer (GPT) incorporates 2 distinct modes—real-time search and pretrained model utilization—it encounters challenges in dealing with these tasks. Specifically, the real-time search can pinpoint some relevant articles but occasionally provides fabricated papers, whereas the pretrained model excels in generating well-structured summaries but struggles to cite specific sources. In response, this study introduces RefAI, an innovative retrieval-augmented generative tool designed to synergize the strengths of large language models (LLMs) while overcoming their limitations.

Materials and Methods

RefAI utilized PubMed for systematic literature retrieval, employed a novel multivariable algorithm for article recommendation, and leveraged GPT-4 turbo for summarization. Ten queries under 2 prevalent topics (“cancer immunotherapy and target therapy” and “LLMs in medicine”) were chosen as use cases and 3 established counterparts (ChatGPT-4, ScholarAI, and Gemini) as our baselines. The evaluation was conducted by 10 domain experts through standard statistical analyses for performance comparison.

Results

The overall performance of RefAI surpassed that of the baselines across 5 evaluated dimensions—relevance and quality for literature recommendation, accuracy, comprehensiveness, and reference integration for summarization, with the majority exhibiting statistically significant improvements (P-values <.05).

Discussion

RefAI demonstrated substantial improvements in literature recommendation and summarization over existing tools, addressing issues like fabricated papers, metadata inaccuracies, restricted recommendations, and poor reference integration.

Conclusion

By augmenting LLM with external resources and a novel ranking algorithm, RefAI is uniquely capable of recommending high-quality literature and generating well-structured summaries, holding the potential to meet the critical needs of biomedical professionals in navigating and synthesizing vast amounts of scientific literature.

Introduction

Biomedical sciences, an ever-evolving and dynamic field, witness an influx of more than 1.5 million papers annually, accumulating over 36 million literature citations in PubMed.^1,2 Researchers in this domain are tasked with the challenging endeavor of thoroughly examining academic publications. Particularly, for literature review, a vital part of scientific research, scholars find it particularly time-consuming.³ Some studies indicate that crafting a literature review paper necessitates a minimum commitment of 6 months.^4,5 In this context, the automatic provision of precise literature recommendation and summarization, grounded in systematic retrieval, is crucial for biomedical professionals.

Generative pretrained transformer (GPT), developed by OpenAI, represents a groundbreaking advancement in artificial intelligence (AI). As a cutting-edge large language model (LLM), GPT has exhibited unprecedented capabilities across an array of natural language processing tasks, including text summarization, dialogue generation, and language translation.^6–10 However, its earlier iterations, including GPT-3.5 and predecessors, faced challenges in delivering authentic literature references.¹¹ Day reported that the academic citations and references provided by ChatGPT, as of February 2023, were often fictitious.¹² Alkaissi and McFarlane¹³ observed that while ChatGPT could compose convincing scientific essays, it mixed both accurate and fabricated references. Sharun et al¹⁴ reported that 15.12% were fabricated and 9.30% were erroneous among the analyzed 86 references. They also observed artificial hallucinations, where the generated text seemed plausible but deviated from fact.¹⁴ To mitigate these issues, Jin et al¹¹ recommended integrating ChatGPT with traditional literature search engines to enhance its capability to deliver comprehensive and accurate high-level summaries. The evolution of ChatGPT continues with the introduction of GPT-4 turbo in November 2023. The latest iteration offers 2 incompatible functionalities: a Bing-based real-time search and the utilization of pretrained LLMs. While the real-time search can pinpoint some latest developments on specific topics with accurate links, it often blends search results with nonresearch papers, like online news, blogs, etc. In contrast, the pretrained model can generate a coherent summary but struggles to provide precise sources. The dichotomy highlights ongoing challenges in LLMs-aided biomedical literature retrieval and recommendation, underscoring the need for further advancements in this task.

In addition to ChatGPT, numerous AI-driven tools offer potential solutions for literature retrieval and summarization. Among these is Google Bard, currently known as Gemini, an advanced conversational chatbot, boasting a comprehensive training regimen that incorporates a wide spectrum of textual and programming knowledge.¹⁵ However, studies by Aiumtrakul et al¹⁵ and McGowan et al¹⁶ pointed out Bard’s limitations in generating accurate citations, with a substantial portion of them being either incorrect or fabricated. ScholarAI, another noteworthy tool, functions as a ChatGPT-integrated plugin. It is designed to streamline the discovery of open-access scholarly content.^17,18 Its primary reliance on Springer-Nature journals, however, may restrict access to materials from various publishers or disciplines.¹⁹ Paper Digest, on the other hand, offers a platform for searching, reviewing, and rephrasing literature without hallucination.²⁰ By forgoing the use of LLMs, it avoids generating unjustified results, but also introduces challenges in semantically interpreting queries.

Retrieval-augmented generation (RAG), initially introduced by Lewis et al, emerges as a promising methodology for bolstering the accuracy and credibility of LLMs, particularly in knowledge-intensive domains.^21,22 Through integrating knowledge from external resources, RAG enables continuous updates along with the assimilation of domain-specific insights. Recently, RAG has gained significant attention, showcasing its potential across various applications. Zakka et al²³ introduced a RAG-based language model, leveraging curated medical resources, to assist in clinical question answering. Ge et al²⁴ devised a liver disease-specific LLM chatbot by incorporating clinical guidelines and patients’ notes. Unlu et al²⁵ presented a RAG-enabled GPT-4 model for clinical trial screening, utilizing clinical notes. Inspired by these efforts, our objective is to create RefAI, a RAG-based tool that harnesses the advanced language comprehension and summarization capabilities of ChatGPT, while augmenting performance through systematic retrieval and ranking from literature databases. In line with Milian et al’s²⁶ advocacy for originality and authenticity in academia, RefAI endeavors to distill verified facts from peer-reviewed publications while harnessing ChatGPT for text summarization, thereby enhancing and streamlining scientific writing for biomedical researchers and clinicians.

Methods

In this study, we utilized PubMed as the primary source to acquire biomedical articles and developed an innovative multivariable algorithm to identify the most pertinent literature on a specific given topic. We deployed the GPT-4 turbo for 3 key purposes: generating keywords for PubMed retrieval based on specific prompts; incorporating the authentic references generated by GPT-4 into the reference pool for further selection; and summarizing recommended literature in response to the queries.²⁷ Our research was conducted on a high-performance server with 8 Nvidia A100 GPUs, each boasting an 80 GB memory. Figure 1 provides an overview of the RefAI framework.

Figure 1.

Overview of the RefAI framework.

Pipeline development

Literature retrieval and metadata acquisition

Literature retrieval and metadata acquisition were pivotal for sourcing relevant articles and their comprehensive details. We employed 2 approaches for accomplishing these, leveraging the capabilities of PubMed and the GPT-4, as well as other literature-related resources (eg, Google Scholar, Journal Citation Reports).

The first approach was to retrieve literature from PubMed. As a widely recognized and freely accessible search engine, PubMed served as our main repository, offering access to over 36 million citations across the fields of biomedicine and health.² We employed the GPT-4 to decompose our initial query—Prompt I—into keywords (Figure 1). Prompt I was structured as “Please extract the keywords from the following research topic, so that they can be sent to PubMed as query words. Please output the keywords directly, splitting by ‘,’: [research topic].” The generated keywords were then sent to PubMed API to fetch pertinent articles along with their metadata, which included the article title, abstract, PMID, authors, publication year, journal, digital object identifier (DOI), and PubMed link. These metadata were considered essential for our subsequent algorithm-driven ranking and manual evaluation processes.

The second approach was to acquire references directly from GPT-4 using Prompt II (Figure 1). Prompt II was designed as “Please recommend ten papers related to [research_topic]. Please output the title and DOI into a list of dictionaries directly. For example: [{‘Title’: ‘Precision Tumor Recognition by T Cells With Combinatorial Antigen-Sensing Circuits’, ‘DOI’: ‘10.1016/j.cell.2016.01.011’}, {…}, …].” This prompt ensured the provision of the article title and DOI, which were then verified through PubMed for authenticity and to gather their metadata.

Beyond the PubMed metadata, we also incorporated citation number and journal impact factor (JIF) to enrich the information of our references. The citation number for the retrieved paper was obtained through a DOI-based query in Google Scholar, using SerpApi.²⁸ JIFs were acquired from the Journal Citation Reports,²⁹ offering insight into the journal’s overall prestige and impact in the field.

Literature recommendation

After merging articles obtained from the above 2 approaches in literature retrieval, we embarked on a series of rigorous selection processes to recommend the top 10 most valuable references from this voluminous candidate pool.

First, we prioritized highly relevant references. Leveraging SentenceTransformer,³⁰ we calculated the article’s relevance score to the query. This transformer converted queries and key reference information (title and abstract) into vector embeddings, enabling a comparison of their cosine similarity. To ensure including the most pertinent references, we set a relatively high threshold of 0.6. Namely, only references with relevance score ≥0.6 were deemed highly relevant and progressed for further consideration.

Second, we imposed a requirement that each study must have an abstract, affirming its research-oriented nature and potential value to scholars.

Third, to rank the retrieved papers, we devised a multivariable algorithm that comprehensively considered relevance, JIF, and citation number, as reflected in eqn (1). The algorithm calculated a total score based on a weighted aggregation of these 3 distinct metrics. It is noteworthy that in this stage, we underscored the significance of JIF and citation score given the high relevance threshold set in the first step.

$$\mathrm{Total}\mathrm{ }\mathrm{score}=\alpha \times \mathrm{relevance}\mathrm{ }\mathrm{score}+\beta \times \mathrm{citation}\mathrm{ }\mathrm{score}+\gamma \times \mathrm{journal}\mathrm{ }\mathrm{impact}\mathrm{ }\mathrm{factor} \mathrm{score}$$ (1)

where $\alpha +\beta +\gamma =1$.

Specifically, for citation score, we employed a piecewise function that integrated the number of citations, publication year, and the current year, as outlined in eqn (2). The function introduced a dynamic scoring mechanism where recent articles, published within the last 2 years, received a maximum citation score of 1. This was to mitigate the inherent bias toward newer articles in citation frequency. Articles older than 2 years without any citations were assigned a minimum score of 0. For all other cases, the citation score was determined by an exponential function, with the age of the article and the number of citations impacting the score inversely and directly, respectively, using Euler’s Number as the base.

$$\text{citation}\hspace{0.17em}\text{score}=\{\begin{array}{ll}1& \text{if}\hspace{0.17em}{y}_c-y_p\le 1\\ 0& \text{if}\hspace{0.17em}\hspace{0.17em}{y}_c-y_p>1\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\hspace{0.17em}c=0\\ e^{-\lambda \left(\frac{y_c-y_p}{c}\right)}& \text{otherwise}\end{array}$$ (2)

where $y_c=\mathrm{current}\mathrm{ }\mathrm{year},\mathrm{ }{y}_p=\mathrm{publication}\mathrm{ }\mathrm{year},c=\mathrm{Google}\mathrm{citation}\mathrm{ }\mathrm{number}$, we take $\mathsf{\lambda }=0.2\mathrm{ }\mathrm{as}\mathrm{ }\mathrm{a}\mathrm{ }\mathrm{use}\mathrm{ }\mathrm{case}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{this}\mathrm{study}$.

To derive the JIF score, we recorded the mean and standard deviation of the impact factors from the gathered journals. The JIF score, as seen in eqn (3), was approximated using the cumulative probability of a normal distribution with a significance level set at 0.05.³¹  

$$\mathrm{journal}\mathrm{ }\mathrm{impact}\mathrm{ }\mathrm{factor}\mathrm{ }\mathrm{score}=\mathrm{ }\Phi \left(z\right)=\frac{1}{\sqrt{2\pi }}{\int }_{-\mathrm{\infty }}^z{e}^{-\frac{z^2}{2}}dz\mathrm{ }\mathrm{ }\left(-\mathrm{\infty }<z<+\mathrm{\infty }\right)$$ (3)

where $z\mathrm{ }=\mathrm{ }\frac{X-\mu }{\sigma }$; X is the journal impact factor, μ is the average impact factor of the collected journals, and σ is the standard deviation of the impact factors of the collected journals.

Furthermore, to determine the optimal weight configuration for eqn (1), 2 core members carefully evaluated a certain number of the results across 10 use case queries (the specific queries are detailed in the subsequent section). Through iterative experiments, we assigned the weights of 0.05, 0.5, and 0.45 to relevance ($\alpha$), citation ($\beta$), and JIF ($\gamma$) scores, respectively. Given the threshold of relevance score set in the first step, the strategic decision—assigning relatively high weights to citation and JIF scores while a relatively low one to relevance score—ensured a prioritized identification of the top 10 studies from prestigious journals and/or enjoying substantial citation acclaim among the highly relevant candidates.

Literature summarization

We employed GPT-4 as the primary model for literature summarization—generating coherent responses to user queries based on the extracted metadata from the recommended articles in HTML format (one example shown in Box S1). The prompt was structured as follows: “Please answer {user_input} coherently and logically from different aspects based on the following HTML data, including the APA in-text citations and a reference list, along with links: {result_dict}.” The prompt provided a structured framework facilitating the system to generate well-organized responses. These responses seamlessly incorporated APA in-text citations, ensuring accurate referencing. Furthermore, a comprehensive reference list followed the summary, offering the complete sources utilized in formulating the responses.

Baselines

Three advanced tools served as baselines for performance comparison. The first, ChatGPT-4, is the latest iteration of the ChatGPT series with 1.76 trillion parameters, representing a significant leap in text generation and conversational capabilities.^32,33 The second is ScholarAI, a dynamic plugin known for its focus on academic content, based on a database of over 200 million scientific peer-reviewed articles, standing out for its specialized ability to handle research-oriented queries and discussions.³⁴ The third is Gemini, Google’s latest innovation in language models.³⁵ In addition, we also examined PubMed.³⁶ Serving as an ablation study, we only investigated PubMed’s literature recommendation capabilities via searching the research topic directly, without incorporating its results for subsequent literature summarization. This focused assessment sought to quantify the added value of RefAI’s multivariate algorithm in enhancing the literature recommendation process.

Query prompts

To evaluate the performance of the baselines, we employed 2 distinct prompts: “Can you recommend 10 research articles related to [research topic], including their titles, journal names, and publication years?” for literature recommendation, and “Can you summarize these articles into a few paragraphs, incorporating in-text citations and providing a reference list?” for literature summarization.

Use cases

We identified 2 representative topics in clinical medicine and informatics, namely “Cancer Immunotherapy and Target Therapy” and “LLMs in Medicine,” as our use cases. For a comprehensive assessment, we extended each main topic into 5 specific subtopics (Table 1).

Table 1.

Two topics with their 5 subtopics as the use case queries.

Topics	Subtopics
1. Cancer immunotherapy and target therapy	Chimeric antigen receptor T-cell therapy for lymphoma
	mRNA cancer vaccine for melanoma
	DNA vaccine for melanoma
	Peptide for melanoma
	Antibody-drug conjugate in the treatment of breast cancer
2. LLMs in medicine	Large language model for medical education
	Performance evaluation of ChatGPT in addressing medical inquiries and issues
	Large language model or ChatGPT in Psychiatry
	ChatGPT for cancer
	ChatGPT for neurosurgery

Abbreviations: LLMs = large language models; mRNA = messenger RNA.

Cancer immunotherapy and target therapy

In recent years, cancer immunotherapy has emerged as a subject of intense interest in oncology, with cancer vaccines standing out as a promising avenue within this field.³⁷ By selecting the topic of “Cancer Immunotherapy and Target Therapy” as a use case, we aimed to delve into the forefront of cancer immunotherapeutics, illuminating its advancements, challenges, and transformative potentials.^38–40

LLMs in medicine

The advent of LLMs has sparked a revolution in medicine and healthcare, harnessing sophisticated AI algorithms trained on unparalleled computing power and multimodal data.^41,42 This use case endeavors to spotlight the myriad applications of LLMs in medicine, tackling challenges head-on and elucidating their transformative influence on healthcare practices.

Evaluation methods

We invited domain experts to evaluate each system. The grading panel comprised 5 medical professionals specializing in cancer immunotherapy and 5 biomedical informatics professionals. Each evaluator was assigned both recommendation and summarization evaluation tasks for 2 subtopics within their expertise. The scoring scale for each dimension was categorized into 5 classes: “1” (poor), “2” (unsatisfactory), “3” (fair), “4” (good), and “5” (perfect). Additionally, for literature recommendation, there was an additional “0” category indicating that the article was not authentic or not relevant at all.

Before the assessment, each evaluator received 2 h of training for the task, including the study objective, the overview of the study workflow, and the evaluation guidelines. To maintain objectivity, the evaluators graded the responses generated independently. Our evaluation criteria for literature recommendation and summarization are shown in Tables S1 and S2, respectively. To mitigate potential rating bias among the evaluators, the final score for each LLM-tool response was determined by averaging the scores from the graders. To facilitate a comprehensive comparison of the tools, we also aggregated the scores in each dimension to derive an overall score for literature recommendation and summarization respectively.

Evaluation of literature recommendation

In the evaluation of the literature recommendation process, we scrutinized 2 key aspects: relevance and quality. The purpose of relevance evaluation was to determine the extent to which the recommended literature aligned with the objectives and scope of the queries. On the other hand, the quality evaluation focused on examining the reliability, credibility, and methodological rigor of each selected article. To facilitate the evaluation for quality, we classified articles into research (eg, original research) and nonresearch (eg, news) types, as detailed in Table S3. Within this framework, we prioritized research-oriented articles, considering them to potentially offer greater intrinsic value to professionals compared with nonresearch ones.

Evaluation of literature summarization

We examined 3 fundamental aspects—accuracy, comprehensiveness, and reference integration—for literature summarization evaluation. Accuracy refers to the precision with which the system effectively captures and conveys the essential information from the source text. It involves the system’s ability to produce summaries that faithfully represent the key concepts, main ideas, and details of the original content without introducing inaccuracies or distortions. Comprehensiveness refers to the extent to which the summary captures a broad range of relevant information and key details from the source text. Reference integration refers to the systematic incorporation of relevant citations, attributions, or connections to external sources within the generated summaries, thereby enhancing the credibility, transparency, and verifiability of the summary.

Statistical analyses

Descriptive statistics, mainly mean and SD, were conducted for the scores provided by evaluators. To assess the agreement between evaluators for each topic, we performed the 2-way intra-class correlation (ICC) analysis. The Shapiro-Wilk test was initially applied to evaluate the normality of the scores provided by different evaluators for the same subtopics. Upon finding the scores deviated from the normality assumption, we opted for the Kruskal-Wallis Rank Sum test to investigate variations among 4 systems. In instances where the Kruskal-Wallis test indicated significant differences, Dunn’s multiple comparison post hoc test was subsequently employed to identify which specific pairs of systems exhibited distinct outcomes. We set a significance threshold of P < .05 for all tests. All statistical analyses were conducted using R software (Version 4.3.2),

We also carried out a quantitative analysis to calculate the components of the recommended articles and proportion of correctly provided metadata (such as title, journal name, and publication year) by each tool during the literature recommendation process.

Results

Evaluation results of literature recommendation

In the context of the literature recommendation, the ICC values revealed a substantial consensus across evaluators when assessing the same subtopic. Specifically, the ICC values spanned from 0.958 to 0.971 for relevance while ranging from 0.919 to 0.944 for quality, indicating strong evaluator concordance (both P-values <.0001). The Shapiro-Wilk test results showed that the distribution of scores for both relevance and quality scores deviated from normality, suggesting that employing a nonparametric approach would be appropriate for investigating significant disparities among the systems evaluated.

In terms of relevance and quality, RefAI notably excelled—achieving the highest mean scores and lowest SD—across all systems (Table 2). Specifically, when evaluating relevance, across 2 main topics which encompassed 10 subtopics in total, RefAI (4.40 ± 1.06) substantially outstripped both ScholarAI (2.93 ± 2.36) and Gemini (1.83 ± 2.30), with P-values <.001. On the quality front, RefAI (4.01 ± 1.21) demonstrated marked superiority over its counterparts—ChatGPT-4 (2.99 ± 1.47), Gemini (1.56 ± 2.02), PubMed (3.43 ± 1.40), and ScholarAI (2.10 ± 1.95), with all P-values <.001. Moreover, when considering an overall score that integrated both relevance and quality, RefAI (8.36 ± 2.17) outperformed all other investigated tools: ChatGPT-4 (7.01 ± 2.47), ScholarAI (5.01 ± 4.13), Gemini (3.31 ± 4.22), and PubMed (7.54 ± 2.41), with all P-values <.01.

Table 2.

Evaluation results of RefAI vs baselines in literature recommendation.^a

Topicsb	Tools	Relevance (0-5)		Quality (0-5)		Overallc (≤10)
		Mean ± SD	P-value	Mean ± SD	P-value	Mean ± SD	P-value
Cancer immunotherapy and target therapy	RefAI	4.36 ± 0.97	–	4.07 ± 1.12	–	8.43 ± 1.94	–
	ChatGPT-4	4.05 ± 1.24	>.05	3.25 ± 1.36	4.57e−04	7.20 ± 2.38	1.48e−03
	ScholarAI	3.02 ± 2.27	5.10e−04	2.35 ± 2.02	4.30e−09	5.38 ± 4.13	2.68e−07
	Gemini	2.21 ± 2.38	2.82e−09	2.00 ± 2.22	1.49e−11	4.21 ± 4.56	2.70e−10
	PubMed	3.95 ± 1.30	>.05	3.52 ± 1.21	6.15e−03	8.43 ± 2.29	6.43e−03
LLMs in medicine	RefAI	4.44 ± 1.16	–	3.93 ± 1.30	–	8.30 ± 2.39	–
	ChatGPT-4	4.09 ± 1.46	>.05	2.69 ± 1.55	7.30e−5	6.79 ± 2.57	5.94e−04
	ScholarAI	2.85 ± 2.44	3.82e−06	1.85 ± 1.88	2.48e−13	4.70 ± 4.12	4.72e−11
	Gemini	1.53 ± 2.19	2.34e−20	1.06 ± 1.70	2.04e−22	2.59 ± 3.79	3.02e−23
	PubMed	4.28 ± 1.31	>.05	3.34 ± 1.57	.05	7.62 ± 2.54	>.05
Overalld	RefAI	4.40 ± 1.06	–	4.01 ± 1.21	–	8.36 ± 2.17	–
	ChatGPT-4	4.07 ± 1.34	>.05	2.99 ± 1.47	2.23e−07	7.01 ± 2.47	3.01e−06
	ScholarAI	2.93 ± 2.36	1.28e−08	2.10 ± 1.95	1.95e−21	5.01 ± 4.13	3.29e−17
	Gemini	1.83 ± 2.30	1.13e−27	1.56 ± 2.02	2.45e−32	3.31 ± 4.22	5.89e−32
	PubMed	4.11 ± 1.31	>.05	3.43 ± 1.40	9.97e−04	7.54 ± 2.41	1.88e−03

Abbreviation: LLMs = large language models.

^a P-value was calculated using Dunn’s post hoc test between RefAI and the baselines.

^bThe total number of scores provided by evaluators for each main topic is 10, regarding each main topic comprising 5 subtopics, each with results from 2 evaluators.

^cThe overall results here represent the aggregate (sum) of the relevance and quality scores.

^dThe overall results here are the mean values from 2 investigated topics.

In addition to the assessments by domain experts, we further examined the article-type composition of literature recommended by different tools (Figure 2). RefAI stood out for its high proportion of research articles (97%) and the absence of fabricated papers. In contrast, ChatGPT-4 exhibited a lower proportion of research articles (43%) and notable percentages of no results (33%) and fabricated papers (3%). Scholar AI and Gemini also showed variations, with Gemini having the highest proportion of fabricated papers (54%). In comparison, PubMed performed similarly to RefAI in research article recommendations (94%). These results underscore RefAI’s strength in recommending sufficient high-quality research articles without fabricated content, distinguishing it from other tools in the study.

Figure 2.

Article type composition of literature recommended by RefAI and baseline tools.

In terms of accuracy of the metadata provided, RefAI and PubMed demonstrated flawless performance of 100% accuracy (Figure 3). In contrast, ScholarAI achieved an accuracy of 80.77%, followed by ChatGPT-4 with 69.23%, and Gemini 62.16%. This highlights the superior capability of RefAI and PubMed in delivering precise metadata information, significantly outperforming their counterparts.

Figure 3.

Accuracy of metadata provided by RefAI and baseline tools.

Evaluation results of literature summarization

For literature summarization, RefAI also exhibited outstanding performance, leading in accuracy (4.40 ± 0.68), comprehensiveness (3.40 ± 1.19), and reference integration (4.60 ± 0.82), as evidenced in Table 3. Specifically, RefAI significantly outperformed ScholarAI and Gemini in terms of accuracy and comprehensiveness (both P-values <.05). Furthermore, in the aspects of reference integration and overall performance, RefAI greatly superseded all 3 baselines (all P-values <.001). These results indicate RefAI’s exceptional ability to generate precise, comprehensive summaries that effectively incorporate references.

Table 3.

Evaluation results of RefAI vs baselines in literature summarization.^a

Tools b	Accuracy (1-5)		Comprehensiveness (1-5)		Reference integration (1-5)		Overall c  (≤15)
	Mean ± SD	P-value	Mean ± SD	P-value	Mean ± SD	P-value	Mean ± SD	P-value
RefAI	4.40 ± 0.68	–	3.40 ± 1.19	–	4.60 ± 0.82	–	12.40 ± 1.64	–
ChatGPT-4	3.55 ± 1.23	.05	3.05 ± 0.88	>.05	2.15 ± 1.31	1.01e−06	8.75 ± 3.05	1.50e−04
ScholarAI	3.10 ± 1.12	1.89e−03	3.05 ± 0.83	.02	3.25 ± 1.41	3.97e−03	9.40 ± 2.58	9.17e−04
Gemini	2.26 ± 1.59	2.67e−06	2.26 ± 1.05	5.13e−04	2.37 ± 1.50	1.99e−06	6.55 ± 3.66	4.53e−08

^a P-value was calculated using Dunn’s post hoc test between RefAI and the baselines.

^bTo enhance the significance of statistical analysis, we merged the results across 10 subtopics, with each being assessed by 2 evaluators, yielding a total of 20 scores for each tool.

^cAggregated results of accuracy, comprehensiveness, and reference integration.

Discussion

In this study, we proposed RefAI, a robust tool designed to enhance LLMs with external literature databases and a novel ranking algorithm. An expert panel of biomedical professionals evaluated the performance of RefAI against established tools across 10 queries spanning 2 pivotal topics. RefAI demonstrated outstanding superiority in biomedical literature recommendation and summarization across 5 critical evaluation metrics (relevance, quality, accuracy, comprehensiveness, and reference integration) over the baseline tools. Table 4 summarizes the strengths and limitations of RefAI and the baseline tools.

Table 4.

Summary of strengths and limitations of RefAI and baseline tools

Tools	Literature recommendation		Literature summarization
	Strengths	Limitations	Strengths	Limitations
RefAI	• Highest relevance and quality • No fabricated articles • Sufficient research papers • High accuracy in metadata provision	• Dependent on PubMed searching mechanism	• Highest accuracy • Highest reference integration	• Comprehensiveness not significantly better than other tools though highest
ChatGPT-4	• High relevance • Relatively low proportion of fabricated articles	• Challenges in offering adequate literature recommendations • High proportion of nonresearch articles • Limitations in retrieving accurate information	• High accuracy • High comprehensiveness	• Poor reference integration
ScholarAI	• Decent accuracy in metadata provision	• High proportion of fabricated articles • Low accuracy in metadata provision	• High comprehensiveness	• Low accuracy
Gemini	• Relatively low proportion of nonresearch articles	• Low relevance and quality • High proportion of fabricated articles • Low accuracy in metadata provision	–	• Unsatisfactory accuracy, comprehensiveness, and reference integration
PubMed	• High relevance • No fabricated articles • Sufficient research papers • High accuracy in metadata provision	• Top-ranked literature may not always be of high quality	• Not applicable	• Not applicable

RefAI distinguishes itself with a precision-driven, multifaceted selection strategy. By leveraging GPT’s advanced language comprehension for keyword extraction, RefAI taps into PubMed’s extensive database for pertinent, credible publications. The integration of a cutting-edge transformer model enables nuanced comparisons between research topics and literature metadata, ensuring highly relevant selections. The system further refines recommendations by considering publication recency, JIF, and citation numbers, thus highlighting influential research. Moreover, RefAI utilizes GPT-4 turbo’s strong text generation capabilities to produce coherent and informative summaries tailored to specific research questions, allowing for sophisticated and comprehensive analyses with deeper insights yielded. In addition, supported by the metadata from PubMed, RefAI seamlessly incorporates citations, facilitating source verification for researchers.

RefAI demonstrates strengths in recommending a sufficient number of authentic articles as requested in the query, characterized by a high percentage of research type and accurate metadata, while standard LLMs encounter certain challenges in these aspects. Specifically, regarding the number of articles recommended, ChatGPT-4 and ScholarAI often yield fewer than the requested 10 works, while Gemini sometimes declines to suggest specific articles, showing insufficient stability. Regarding article type, ChatGPT has limitations in literature recommendation due to its restricted sources. Additionally, ChatGPT 4’s ability to extract only web information occasionally leads to the inclusion of nonresearch materials like scientific news, as many research papers are not easily accessible on journal websites. Concerning the accuracy of metadata and the authenticity of articles, RefAI and PubMed stand out for their exemplary performance. In contrast, Gemini has been observed to suggest a considerable number of fabricated papers. This discrepancy could be attributed to the fact that generative language models like ChatGPT and Gemini might not have undergone extensive training on specialized biomedical literature databases. Consequently, they were not able to accurately provide authentic articles and precise metadata. Moreover, the underlying algorithms and training data may also influence the occurrence of inaccuracies in metadata and the proportion of fabricated papers suggested.

When it comes to domain experts’ evaluations, RefAI exhibited superior performance in terms of literature recommendation. It achieved the highest mean scores and the lowest SD for both relevance and quality metrics. It is noteworthy that in terms of relevance, RefAI didn’t show significantly better performance than PubMed or ChatGPT-4. This observation aligns with expectations, given PubMed’s effective “Best Match” sorting strategy and ChatGPT-4’s advanced language comprehension capabilities. However, RefAI distinctly outperformed both PubMed and ChatGPT-4 in quality. This advantage is attributed to its refined ranking mechanism, which mitigates the inclusion of nonresearch articles such as editorials and comments (normally included in PubMed) or news articles (potentially recommended by ChatGPT-4).

Regarding literature summarization, the highlight of RefAI lies in reference integration, among 3 evaluation metrics. Built on top of PubMed, RefAI enriches its summarization with precise in-text citations and reference lists. In accuracy and comprehensiveness, RefAI’s performance was on par with that of ChatGPT-4. This is reasonable given that RefAI’s summarization capability is also powered by GPT-4. However, RefAI’s overall summarization performance significantly eclipsed that of all baseline tools (P-values <.001). This evidence underscores RefAI’s effective use of GPT-4 for summarization based on well-recommended references.

Despite the strengths of our system, certain limitations exist. First, the dependency on PubMed’s keyword-based retrieval mechanism may restrict its ability to achieve deep semantic comprehension of the queries and potentially miss some important papers.⁴³ Second, a comprehensive examination of various embedding models in literature recommendation should be undertaken to explore potential improvements in performance for this task. Third, we used Likert-scale for manual evaluation, which is crucial for assessing system performance but can potentially bias participant responses.⁴⁴ Moving forward, we plan to develop benchmarks for systematic evaluation to conduct more comprehensive and standardized assessments of literature recommendation systems. Moreover, the token limit of GPT-4 turbo constrains our system’s ability to process large volumes of articles. Further enhancements of OpenAI’s GPT token limits could potentially improve its advantages in comprehensiveness. Lastly, while GPT-4 turbo represents a technological advancement, its generation of fictitious recommendations poses integration challenges. However, we have implemented the innovative theoretical framework and are anticipating OpenAI’s accessibility for developers to retrieve literature.

Conclusion

Our study proposes RefAI, an innovative tool engineered to harness the strengths of ChatGPT while addressing its current limitations by incorporating external databases. RefAI excels in systematically identifying and recommending pivotal literature, coupled with the provision of coherent, well-structured summaries, plus accurate in-text citations and reference lists. The synergistic fusion of ChatGPT with conventional literature databases empowers researchers with streamlined access to and synthesis of relevant literature. At the forefront of an LLMs-powered scientific paradigm shift, RefAI holds promise to help accelerate biomedical research by offering a more efficient approach to literature review and scientific writing.

Author contributions

Yiming Li and Jeff Zhao are co-first authors.

Cui Tao, Fang Li, Jeff Zhao, and Yiming Li designed the study. Yiming Li and Jeff Zhao developed the pipeline. Manqi Li and Yiming Li conducted statistical analysis, and Manqi Li, Yifang Dang, and Yiming Li performed visualization. Yifang Dang, Jianfu Li, Zenan Sun, Jingna Feng, Zehan Li, Fang Li, Evan Yu, Ahmed M. Abdelhameed, and Yiming Li offered technical resource support. Yifang Dang, Wei Tao, and Jianping He contributed to data collection. Zenan Sun, Evan Yu, Daowei Yang, Usama Hussein, Jianfu Li, Jianguo Wen, Ahmed M. Abdelhameed, Junhua Mai, Xinyue Hu, Shenduo Li, Yanyan Lou, Yue Yu, and Jianping He evaluated system performance, with Evan Yu, Tiehang Duan, Yifang Dang, Manqi Li, Jingna Feng, Zehan Li, and Ahmed M. Abdelhameed participating in data management. Yiming Li, Junhua Mai, Jianguo Wen, Fang Li, Ahmed M. Abdelhameed, and Usama Hussein designed the use cases and queries. Yiming Li, Manqi Li, and Fang Li drafted the manuscript. Cui Tao and Fang Li supervised the study and critically revised the manuscript.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This work was supported by the National Institute of Allergy and Infectious Diseases of the National Institutes of Health grant numbers R01AI130460 and U24AI171008, the American Heart Association grant number 19GPSGC35180031, and Cancer Prevention and Research Institute of Texas award RP220244.

Conflicts of interest

The authors have no competing interests to declare.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.

Code availability statement

The code underlying this article will be shared on reasonable request to the corresponding author.