Assessing Large Language Models for Early Article Identification in Otolaryngology—Head and Neck Surgery Systematic Reviews

ABSTRACT

Background

Assess ChatGPT and Bard's effectiveness in the initial identification of articles for Otolaryngology—Head and Neck Surgery systematic literature reviews.

Methods

Three PRISMA‐based systematic reviews (Jabbour et al. 2017, Wong et al. 2018, and Wu et al. 2021) were replicated using ChatGPTv3.5 and Bard. Outputs (author, title, publication year, and journal) were compared to the original references and cross‐referenced with medical databases for authenticity and recall.

Results

Several themes emerged when comparing Bard and ChatGPT across the three reviews. Bard generated more outputs and had greater recall in Wong et al.'s review, with a broader date range in Jabbour et al.'s review. In Wu et al.'s review, ChatGPT‐2 had higher recall and identified more authentic outputs than Bard‐2.

Conclusion

Large language models (LLMs) failed to fully replicate peer‐reviewed methodologies, producing outputs with inaccuracies but identifying relevant, especially recent, articles missed by the references. While human‐led PRISMA‐based reviews remain the gold standard, refining LLMs for literature reviews shows potential.

The search criteria for three systematic reviews were replicated using ChatGPTv3.5 and Bard. Outputs included author, title, publication year, and journal and were cross‐referenced with medical databases and compared to the original reviews. Large language models failed to replicate peer‐reviewed methodologies.

Abbreviations

AI

artificial intelligence

CV

computer vision

DL

deep learning

FN

false negative

FP

false positive

LLM

large language models

ML

machine learning

NLP

natural language processing

OHNS

otolaryngology—head and neck surgery

TP

true positive

1. Background

Artificial intelligence (AI) is an overarching system whereby computers and machines mimic the human mind using tools such as machine learning (ML), deep learning (DL), natural language processing (NLP), and computer vision (CV) [1]. Each of these algorithms supports AI via data representation to model the brain (DL), correction learning from human interactions (ML), processing of text and language (NLP), and interpretation of visual data points (CV) [2, 3, 4, 5]. Together, these tools have allowed for widespread adoption of AI in various disciplines, including science, medicine, and research via large language models (LLMs), DL models trained on enormous datasets (e.g., OpenAI's ChatGPT and Google's Bard) [6, 7, 8, 9]. LLMs have widespread potential, including automating tasks within systematic reviews [10, 11].

Completing a systematic review has been estimated to require an average of five researchers and 67.3 weeks, with medical librarians spending an average of 26.9 h on search strategies [12, 13]. Overall, the process costs about $141,191.80 per systematic review with meta‐analysis, potentially exceeding $16 million annually for some institutions, making this a time‐ and finance‐consuming endeavor [14]. To evaluate the ability of LLMs to streamline the time‐ and cost‐intensive systematic review process, Issaiy et al. [15]. demonstrated that ChatGPT could screen 1198 abstracts in an hour, in contrast to the 7–10 days typically required by general physicians [15].

To further expand upon these results, our study aimed to compare the effectiveness of two commonly used LLM chatbots, ChatGPT and Bard, to correctly identify articles for inclusion in systematic reviews within the field of Otolaryngology (OHNS) compared to three PRISMA‐compliant systematic reviews. We compare their performance with established PRISMA‐compliant review articles to assess their potential for streamlining the initial stage of systematic reviews.

2. Materials and Methods

2.1. Article Selection

PubMed was reviewed for PRISMA‐guided systematic reviews published in OHNS journals, referencing at least 20 articles, and published before 2022, aligning with ChatGPTv3.5 (ChatGPT) training data cutoff in 2021. Although several systematic reviews fulfilled these criteria, only three of the systematic reviews were selected to showcase this study's proof of methodology. The selected reference reviews were published in reputable OHNS journals and focused on topics such as surgical outcomes, healthcare disparities, demographic analysis, or biomedical applications and include citations of more than 20 qualifying articles. The selected reviews include: (1) “Applications of augmented reality in otolaryngology: a systematic review” authored by Wong et al. 2018 [16]; (2) “Healthcare disparities in pediatric otolaryngology: a systematic review” authored by Jabbour et al. in 2017 [17]; and (3) “The prevalence of olfactory and gustatory dysfunction in COVID‐19—a systematic review” authored by Wu et al. 2021 [18].

2.2. ChatGPT and Bard Query

Collected data from the systematic reviews included authors' names, titles, publication years, journal names, and search terms for implementation into the LLMs via three different prompts. ChatGPT and Bard were selected for their accessibility as free platforms and their simple, chat window interfaces. These queries were designed to simulate straightforward search phrases (Prompt 1) as well as progressively refined search phrases (Prompts 2–3), accounting for prompt engineering considerations (Table S1). For Prompt 1, the query was: “Find published papers regarding ‘search terms’ from/until [desired time‐range].” For Prompt 2, the query was: “You are writing a review article; provide a list of published papers between [desired time‐range], including author names, paper titles, journals, and publication year based on the following topics: ‘search terms’.” For Prompt 3, the query was: “You are writing a review article, provide a list of papers with author names, paper titles, journal, and publication year based on the topic that will follow: ‘search terms’.”

The queries for each article's prompts were executed simultaneously within a single chat window, which remained open within each prompt. However, it was refreshed after each new prompt. To reduce bias and maintain consistency of LLM results, two independent reviewers queried ChatGPT using the same three prompts once each. The queries for Jabbour et al. were run on July 8, 2023, and for Wong et al. and Wu et al. they were run between October 14 and 15, 2023. All gathered information from the LLMs was referred to as “outputs.” All outputs generated by ChatGPT in response to Prompts 1, 2, and 3 were documented and referred to as ChatGPT‐1–ChatGPT‐3, respectively.

The same methodology as the aforementioned ChatGPT queries was replicated for Bard. Subsequently, all outputs generated by Bard, in response to Prompts 1 and 2, were named Bard‐1 and Bard‐2, respectively.

2.3. Outcomes

The outputs from the ChatGPT and Bard queries were validated via searching for the articles in PubMed, with search results showing either the correct author names or article titles being reported as “Genuine.” Results that correctly matched author names with the appropriate article titles were called “Authentic.” “Unique outputs/articles” were those included after duplicated outputs were excluded. Furthermore, the same inclusion and exclusion criteria from the selected reference articles were employed to identify outputs that may have been missed by the reference articles (Figure 1).

Figure 1.

PRISMA‐like breakdown of LLM outputs. The figure shows the breakdown of outputs by ChatGPT and Bard (a–c) for queries related to the reference articles Wong et al. Jabbour et al. and Wu et al., respectively.

Authentic outputs that met screening criteria and were included in the reference reviews were identified as true positives (TPs). False positives (FPs) were authentic outputs that did not meet screening criteria. False negatives (FNs) were authentic article outputs missed by the reference review articles, yet met the reference's inclusion criteria. True negatives were authentic article outputs not included by the chatbot or the systematic review. Since we could not account for this, this value was 0 throughout. “Recall” is defined as the ability of the chatbot to identify articles that match the reference from all relevant articles identified. Recall percentages were calculated by dividing TPs by the sum of TPs and FPs, multiplying by 100, and were calculated for each review's prompt (Table 1).

Table 1.

Recall analysis of LLM outputs. Table showing the values for true positives, false positives, true negatives, false negatives, and the recall percentages for ChatGPT and Bard (a–c) for queries related to the reference articles.

Chatbots	True positives (TPs)	False positives (FPs)	True negatives (TNs)	False negatives (FNs)	Recall (%)
Wong et al.
ChatGPT‐1	0	0	0	0	0
ChatGPT‐2	0	0	0	0	0
ChatGPT‐3	0	1	0	0	0
Bard‐1	0	0	0	0	0
Bard‐2	2	2	0	0	100
Jabbour et al.
ChatGPT‐1	0	2	0	1	0
ChatGPT‐2	0	0	0	0	0
ChatGPT‐3	0	3	0	0	0
Bard‐1	0	14	0	0	0
Bard‐2	0	1	0	0	0
Wu et al.
ChatGPT‐1	0	0	0	0	0
ChatGPT‐2	6	3	0	4	60
ChatGPT‐3	7	11	0	13	35
Bard‐1	0	2	0	2	0
Bard‐2	0	3	0	0	0

Finally, bar graphs were constructed to represent variability in publication years visually and journals between the review articles and the generated outputs (Figures 2, 3, 4, 5, 6).

Figure 2.

Comparison of ChatGPT and Bard outputs to reference articles. The figure compares the outputs generated by ChatGPT (a–c) and Bard (d–f) with those from the reference articles Wong et al., Jabbour et al., and Wu et al.

Figure 3.

Comparison of ChatGPT and Bard genuine outputs to reference articles. The figure compares the genuine outputs generated by ChatGPT (a–c) and Bard (d–f) with those from the reference articles Wong et al. Jabbour et al. and Wu et al.

Figure 4.

Top journals represented in ChatGPT and Bard outputs compared to reference articles. The figure compares the top journals found in the outputs generated by ChatGPT (a–c) and Bard (d–f) with the reference articles Wong et al. Jabbour et al. and Wu et al.

Figure 5.

Publication date distribution in ChatGPT and Bard outputs compared to reference articles. The figure compares the distribution of publication dates in the outputs generated by ChatGPT (a–b) and Bard (c–d) with the reference articles Wong et al. and Jabbour et al.

Figure 6.

ChatGPT non‐response examples for Prompt 1. The figures illustrate the various responses provided by ChatGPT when it failed to generate any outputs for Prompt 1 (a–c).

3. Results and Analysis

3.1. Comparative Analysis of ChatGPT and Bard in Replicating Search Outputs From Reference Articles

Wong et al. initially identified 252 articles but included only 23 after excluding duplicates and applying inclusion/exclusion criteria. Similarly, Jabbour et al. started with 969 articles and ended with 61, while Wu et al. began with 198 but only included 25 articles (Figure 2a–f).

When the study team replicated reference article search criteria, ChatGPT produced 0, 39, and 55 outputs for Prompts 1, 2, and 3, respectively, for Wong et al. (Figure 2a). One duplicate was found in the ChatGPT‐3 dataset, reducing the number of outputs to 54. For Jabbour et al., ChatGPT produced 20, 62, and 41 outputs for Prompts 1, 2, and 3, respectively (Figure 2b). ChatGPT‐2 and ChatGPT‐3 had 1 and 3 duplicates, reducing the unique outputs to 61 and 38, respectively. Finally, for Wu et al., ChatGPT produced 0, 83, and 134 outputs for Prompts 1, 2, and 3, respectively (Figure 2c). After removing duplicates, there were 19 and 37 unique outputs for Prompts 2 and 3, respectively.

Bard generated 81 and 280 outputs for Prompts 1 and 2 in the Wong et al. study, with 37 and 47 unique outputs after removing duplicates (Figure 2d). For Jabbour et al., Bard generated 85 and 92 outputs for Prompts 1 and 2, with 36 and 35 unique outputs after removing duplicates (Figure 2e). Finally, in the Wu et al. study, Bard generated 259 and 498 outputs for Prompts 1 and 2, with 33 and 35 unique outputs after removing duplicates, respectively (Figure 2f).

3.2. Comparison of ChatGPT and Bard Performance in Generating Genuine References

The Wong et al. study results showed that ChatGPT‐2 identified no genuine article titles and 6 genuine authors, while ChatGPT‐3 identified 11 and 7 genuine article titles and authors, respectively, with only 1 being authentic (Figure 1a). Despite this, none of the prompts produced outputs matching the articles referenced in Wong et al. (Figure 1a). In the Jabbour et al. study, ChatGPT‐1 identified 3 genuine article titles and 4 genuine authors, ChatGPT‐2 identified 1 genuine article title and 61 genuine authors, and ChatGPT‐3 identified 5 genuine article titles and 4 genuine authors. Of these, 2, 0, and 3 were authentic articles, respectively (Figure 1b), with none matching Jabbour et al.'s references (Figure 3b). For the Wu et al. article, ChatGPT‐2 generated 14 genuine article titles and authors, while ChatGPT‐3 generated 33 and 34 genuine article titles and authors, respectively, with 7 matching authentic references from Wu et al. compared to 6 from ChatGPT‐2 (Figure 3c). Additionally, ChatGPT‐2 and ChatGPT‐3 identified 4 and 13 authentic articles, respectively, that Wu et al.'s inclusion and exclusion criteria missed in the systematic review.

For Bard, Bard‐2 had 9 and 4 genuine article titles and authors, respectively, while Bard‐1 had 3 genuine article titles and no genuine authors (Figure 3d). Bard‐2 identified 4 authentic articles; 2 matched those in the reference article by Wong et al. and the other 2 met the inclusion/exclusion criteria but were missed by the systematic review. Bard‐1 did not identify genuine articles matching the reference (Figure 1a).

Genuine article titles and author lists for Bard‐1 were 14 and 36, and for Bard‐2, 7 and 18, respectively, in the Jabbour et al. study (Figure 3e). Bard‐1 and Bard‐2 identified 14 and 1 authentic articles, respectively. Yet, none of these articles matched the references or followed the inclusion/exclusion criteria defined by the reference Jabbour et al. (Figure 1b).

In the Wu et al. study, Bard‐1 and Bard‐2 had 4 and 5 genuine article titles, 5 and 6 genuine authors, and 4 and 3 authentic articles, respectively, none of which matched the citations from the review article (Figure 3f). However, Bard‐1 produced 2 articles that met the inclusion/exclusion criteria but were missed by the Wu et al. study (Figure 1c).

3.3. Journal Preferences and Publication Year Distribution in Outputs Generated by ChatGPT and Bard

For the output of Wong et al. ChatGPT‐2 and ChatGPT‐3 preferred the Otolaryngology—Head and Neck Surgery Journal (OHNSJ) (10% of outputs), unlike the reference article, in which the most cited journal was Studies in Health Technology and Informatics (17%) (Figure 4a). In the Jabbour et al. study, The Laryngoscope (18%) was the most referenced journal, whereas OHNSJ dominated ChatGPT outputs (Figure 4b). OHNSJ predominated in Wu et al. (32%) regarding journal representation, whereas, for ChatGPT‐2, 9.1% of outputs were published in the International Forum of Allergy and Rhinology (IFAR), OHNSJ, Chemical Senses, Nature Medicine, and The Laryngoscope, each. Alternatively, ChatGPT‐3 displayed a tie among three journals, with 9.8% of outputs published in Head and Neck, IFAR, and European Archives of Oto‐Rhino‐Laryngology (EAORL) (Figure 4c).

For the most commonly cited journal in Wong et al. output, 14.3% of Bard‐1's outputs were published in Laryngoscope Investigative Otolaryngology (LIO), while 17.4% of Bard‐2's were in the Journal of Otolaryngology—Head and Neck Surgery (JOHNS) (Figure 4d). The most cited journal for Bard was The Laryngoscope (25% for Bard‐1 and 17% for Bard‐2) for the Jabbour et al. output (Figure 4e). In Wu et al. the most common journal was Lancet Infectious Diseases (37.8%) for Bard‐2, while Bard‐1's was a tie between OHNSJ and BMJ Case Reports (20%) (Figure 4f).

For the Wong et al. query, ChatGPT‐2's outputs were exclusively from 2010 to 2019, while ChatGPT‐3 displayed more variation with 1.9%, 79.6%, and 18.5% of outputs published between 2000 and 2009, 2010 and 2019, and 2020 and 2023, respectively. In the reference article, Jabbour et al. selected manuscripts from 1990 to 2019, predominantly from 2000 to 2009 (27.9%) and 2010 to 2019 (63.9%) (Figure 5a). In contrast, ChatGPT‐1 focused exclusively on 2010–2019 (100%) for its Jabbour et al. outputs. ChatGPT‐2 had 46.9% and 53.2% of its output from 2000 to 2009 and 2010 to 2019, respectively, while ChatGPT‐3's output was exclusively from 2010 to 2019 (51.2%) and 2020 to 2023 (43.9%) (Figure 5b).

Most of Bard‐2's outputs (97.8%) for Wong et al. were from 2010 to 2019, with one article from 2022, while all of Bard‐1's outputs were from 2010 to 2019 (Figure 5c). For Jabbour et al. 55% of Bard‐1's outputs were from 2010 to 2019, whereas only 28.6% of Bard‐2's outputs were from 2010 to 2019, with the largest portion (48.6%) being from 2020 to 2023. Of note, 22.2% of Bard‐1's and 8.6% of Bard‐2's outputs were from 1980 to 1989; however, Jabbour et al. did not include any articles from this period (Figure 5d).

3.4. Performance Comparison of ChatGPT and Bard in Identifying TP, FN, and FP Across Different Datasets

The number of TP for ChatGPT‐2 was greater than that of Bard‐2, as none of the outputs identified by Bard‐2 matched those of the reference, while 6 outputs from ChatGPT‐2 matched the reference article. Additionally, ChatGPT‐2 had a greater number of FNs, identifying 4 articles missed by the reference Wu et al. (Figure 1c). On average, Bard generated more FP than ChatGPT in the Jabbour et al. dataset (Table 1). While all models failed to exhibit recall for Jabbour et al. Bard‐2 achieved perfect recall for the Wong et al. output, and ChatGPT‐2 had 60% recall for the Wu et al. outputs (Table 1).

4. Discussion

LLM chatbots, like OpenAI's ChatGPT and Google's Bard, have greatly enhanced public access to and awareness of AI and offer potential research applications [9, 19, 20, 21]. The role of AI in healthcare research has increased tremendously in recent years, benefiting the range of medical research from case reports to systematic reviews in fields such as oncology, reproductive medicine, and neurosurgery [15, 22, 23, 24, 25, 26]. Though several medical/research‐focused LLMs have emerged, the most commonly accessed LLMs are free, general knowledge chatbots that have not been thoroughly evaluated for their utility in performing the initial stage of a systematic review.

As such, this study investigated the effectiveness of ChatGPT and Bard in identifying relevant articles. We employed three prompts to assess ChatGPT's accuracy in identifying OHNS articles. The prompts were created to emulate users with varying levels of technical expertise. Prompt 1 was a search option for novice users of chatbots and systematic reviews, whereas Prompts 2 and 3 modeled more advanced searches, mimicking PubMed with MeSH terms and Boolean operators for more precise article selection. Regarding Bard's prompts, only a single beginner and a single advanced search option were employed.

The results revealed several key themes. Prompt 1 generated the fewest outputs for both chatbots, with ChatGPT sometimes failing to produce a single outcome (Figure 6). Additionally, Bard consistently produced more outputs per query compared to ChatGPT's observed limitation of five outputs (Figures 1 and 2). Bard also produced some outputs outside specified time frames and occasionally extended beyond user‐defined queries (Figure 5).

Our analysis revealed mixed performance by ChatGPT and Bard in identifying articles matching those included in the reference studies. While both models retrieved some authentic articles, many did not align with the selection criteria established by the respective reference studies (Jabbour et al. Wong et al. and Wu et al.) (Figure 1). Notably, ChatGPT‐1 retrieved one relevant article missing from the Jabbour et al. (Figure 1b), while Bard‐2 generated articles consistent with Wong et al. (Figure 1a). In the Wu et al. studies, most ChatGPT models (except ChatGPT‐1) produced authentic outputs, with ChatGPT‐2 and ChatGPT‐3 even identifying articles not included in the original study (Figure 1c). Bard‐1 also retrieved some relevant articles missed by Wu et al. though to a lesser extent than ChatGPT‐2 and ChatGPT‐3. Interestingly, the Wu et al. prompts yielded the most relevant outputs overall (signified by higher TPs and FNs).

Further comparisons showed Bard outperformed ChatGPT in terms of total outputs generated (Figure 2a–c). However, Bard produced more duplicates, while ChatGPT was more consistent in identifying authentic articles. For the Wu et al. paper, ChatGPT‐2 identified 13 authentic articles, compared to just 3 by Bard‐2, despite Bard generating fewer outputs. Bard also demonstrated a broader range of publication years, retrieving outputs from as early as the 1970s in the Jabbour et al. study, whereas ChatGPT‐1 only generated outputs from 2010 to 2019. Bard‐2's results were more diverse, including articles 2020–2023, which ChatGPT‐2 did not capture as effectively (Figure 5b,d). As of July 2025, a brief search using the most up‐to‐date versions of these chatbots (ChatGPT‐4o and Gemini 2.5Pro) showed similar results overall. Gemini (formerly Bard) generated more outputs than ChatGPT; however, ChatGPT identified more genuine articles and retrieved articles that matched the reference articles and met the inclusion criteria (Figure S1).

These findings suggest that ChatGPT may require more user expertise for effective query formulation or may have more internal guards to prevent spurious results. In contrast, Bard demonstrated greater user‐friendliness by generating some outputs even with simpler prompts. The reason behind ChatGPT's limitation to five outputs per query remains unclear, but it could stem from its software developer's set limits. While it is possible to prompt ChatGPT for additional results, Bard's ability to provide more accurate and larger quantities of outputs indicates that it may be designed for more lenient queries. In systematic reviews, a comprehensive output is crucial to identify all relevant information, and ChatGPT's limited output generation poses a limitation. Both Bard and ChatGPT could be improved to balance comprehensiveness with adherence to search parameters.

When further evaluating article retrieval for the Wu et al. paper, the high yield may be due to the recency of publication. This could signify an improvement in LLMs' article identification capabilities. Another potential explanation could be that the articles included in Wu et al. were required to be open access, which may have assisted chatbots in better identifying articles that were not behind a subscription paywall [18].

Interestingly, outputs from both chatbots identified genuine articles or authors on PubMed; however, article titles often contradicted author lists, and content sometimes lacked relevance to the queries. Additionally, outputs frequently included content unrelated to OHNS, evidenced by the FPs (Table 1). Furthermore, both Bard and ChatGPT exhibited “artificial hallucinations,” generating false information in response to queries, which was more prevalent with Bard in our study (Figure S2). A recent search using the most updated versions of ChatGPT‐4o and Gemini 2.5Pro indicates that artificial hallucination still occurs. Although sometimes subtle, these hallucinations continue to pose challenges for research (Figure S3). This phenomenon has been highly reported in the literature across several LLMs and is attributed to reinforcement learning and the lack of definitive truth sources [27, 28]. While ongoing efforts aim to minimize errors in each LLM iteration, LLM‐generated outputs should still be verified [29].

While LLMs excel at prediction‐based text generation, they do not actively search for real sources [11]. Systematic reviews, however, rely on verifiable, evidence‐based information. AI‐generated misinformation poses a threat to infiltrating this trusted system and could have detrimental consequences for healthcare due to distrust and potentially harmful outcomes. Additionally, training data bias can affect the quality of review processes and outcomes, as evidenced by Bard and ChatGPT's inability to accurately identify and match articles from the reference lists. These mismatches could be partially attributed to biases in the training data used for these LLMs. Furthermore, the inherent variability in LLM outputs—different results from the same prompt—presents a significant hurdle. Current literature review standards often require multiple reviewers to identify articles, which would be undermined if other users cannot reproduce the LLMs' results.

Our study further revealed other limitations associated with the training period of LLMs and the time frame of outputs. ChatGPT's output exhibited a bias toward publications from 2010 to 2023 (Figure 4a,b), suggesting a limitation in its training data compared to Bard, which retrieved articles from a wider time frame, including the 1980s. This highlights a concern about LLMs' ability to perform an adequate literature search, as restricted training periods may lead to reliance on outdated information or recency bias, thus presenting a skewed understanding of topics. For LLMs to become practical tools for systematic reviews, they require regular updates to incorporate the latest research and information found offline. Many foundational articles, particularly older print publications, could be missed by LLM searches and must be addressed.

While the current iterations of ChatGPT and Bard cannot be solely relied upon for accurate article identification in systematic reviews, they hold promise for automating literature screening and data extraction. Specialized generative AI tools like Covidence, DistillerSR, PICO Portal, Rayyan, and RobotReviewer have been specifically developed for these tasks [30, 31, 32, 33, 34]. Efforts are also underway to integrate and develop interoperable extensions between various systematic review tools to further automate the process [35]. Notably, this study does not focus on using recent AI tools, such as Elicit and Semantic Scholar, explicitly developed for systematic reviews [36, 37, 38].

A significant challenge in this study was prompt engineering for the LLMs. We employed recommended strategies, including specific requests, role‐playing, threads, and temporal awareness. Prompts 2 and 3 outperformed Prompt 1 for ChatGPT, with ChatGPT‐3 producing the most matches with Wu et al.'s review (Figure 1c). Similarly, Bard‐2 yielded better results than Bard‐1 (Table 1), highlighting the critical role of prompt design in shaping the quantity and quality of LLM outputs and posing a substantial limitation by requiring proficiency in query modification. Future studies should explore alternative techniques, such as providing examples or requesting LLMs to generate effective prompts.

Another limitation is the potential bias due to the evolution of chatbots, as evidenced by the lack of output by ChatGPT for certain queries. In our study, prompts were run using the same window following recommendations to maintain threads and to be processed from the same context as the original query [39]. A study using Elicit for systematic reviews screening found that, under identical conditions but at different times, the AI research assistant produced inconsistent results [38]. This highlights the potential impact of timing on prompt performance, an area that warrants further investigation in future research. However, this study sought to account for this by having two independent reviewers conduct the same searches simultaneously. Future studies could mitigate this bias by conducting simultaneous searches with different LLMs, resetting the chat box after each search to minimize the influence of previous interactions, and repeating the same search prompt multiple times to determine whether additional relevant papers are identified in subsequent iterations. Recent updates to ChatGPT include guidance on conducting literature reviews using established databases and showcasing advancements with safeguards against misinformation (Figure S4). OpenAI's latest update with ChatGPT 4 Turbo, Custom GPT, and Assistants API further expands the potential applications of LLMs [35]. Recent studies have shown that ChatGPTv4.0 API script, when prompted with inclusion and exclusion criteria, achieved 96% specificity and 93% sensitivity in screening even without fine‐tuning [40], highlighting the potential collaborative role these tools may play in future research workflows.

Finally, the prompt design of our study primarily based on date range and search terms deviates from the structured and comprehensive approach of a traditional systematic review. Our methodology captures only a limited component of the broader review process, and as such, lacks the depth and comprehensiveness required for a complete review. This is an important limitation, particularly in the early stages of literature identification, where direct use of academic search engines or specialized AI tools such as Covidence, DistillerSR, PICO Portal, Rayyan, and RobotReviewer would typically yield a broader and more systematic set of articles. Nevertheless, given the rapid adoption of chatbots, it is important to evaluate their potential role and define their place within scientific workflows such as systematic reviews.

5. Conclusion

To our knowledge, this study represents the first comparative analysis of Bard and ChatGPT in OHNS research exploration. Our findings suggest that Bard and ChatGPT are not suitable as primary tools for identifying articles for a systematic review, though LLMs may serve a supplementary role in the systematic review process [11, 41, 42, 43]. This work serves as a cautionary tale of incorporating general knowledge LLMs for medical or scientific usage, particularly for young researchers, researchers inexperienced with AI, and the general public. We advocate relying on standard protocols and established databases for more reliable and valid systematic review results. Further research should explore methods to integrate LLMs into the review process, potentially by improving search accuracy, incorporating human oversight for validation, and tailoring prompts to specific research questions. By addressing these limitations and leveraging their strengths, LLMs may evolve into valuable assets for OHNS researchers conducting efficient and comprehensive literature reviews.

Author Contributions

Ajibola B. Bakare: conceptualization (equal), data curation (equal), formal analysis (equal), investigation (equal), methodology (equal), validation (equal), visualization (equal), writing – original draft (equal), writing – review and editing (equal). Young Lee: conceptualization (equal), data curation (equal), formal analysis (equal), investigation (equal), methodology (equal), validation (equal), visualization (equal), writing – original draft (equal), writing – review and editing (equal). Jhuree Hong: conceptualization (equal), methodology (supporting), validation (equal), writing – review and editing (supporting). Claus‐Peter Richter: investigation (supporting), methodology (supporting), supervision (supporting), writing – review and editing (supporting). Jonathan P. Kuriakose: conceptualization (lead), investigation (equal), methodology (equal), project administration (equal), supervision (lead), validation (equal), visualization (equal), writing – review and editing (equal).

Funding

The authors have nothing to report.

Ethics Statement

The authors have nothing to report.

Consent

The authors have nothing to report.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Supplemental Figure 1: Outputs from ChatGPT‐4o and Gemini 2.5Pro (formerly Bard) in response to queries on reference articles: a) Wong et al., b) Wu et al., and c) Jabbour et al., based on a follow‐up search conducted June 4‐11, 2025.

Supplemental Figure 2: Illustration of artificial hallucinations in Bard‐1 (a) and Bard‐2 (b). A comparison with ChatGPT‐1 and ChatGPT‐2 (c) underscores the limitations and inclination toward generating fictitious outputs.

Supplemental Figure 3: Artificial hallucination examples from ChatGPT‐4o and Gemini 2.5Pro (June 4‐11, 2025). a) Top/middle/bottom panels: ChatGPT‐4o output, linked article, and actual PubMed article. b) Top/bottom panels: Gemini 2.5Pro output and actual PubMed article. All details match except author names. Supplemental Figure 4: A re‐run of ChatGPT‐2 on 11/23/23, showcasing the implem entation of new safeguards preventing the generation of article outputs.

Supplemental Table 1: Comprehensive methods and search terms utilized in querying ChatGPT and Bard.

Bakare A. B., Lee Y., Hong J., Richter C.‐P., and Kuriakose J. P., “Assessing Large Language Models for Early Article Identification in Otolaryngology‐Head and Neck Surgery Systematic Reviews,” Health Care Science 0 (2026. 5:): 19–28. 10.1002/hcs2.70048.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.