Large language models vs. conventional search in identifying delirium guidelines for a scoping review

Abstract

Background

A literature search of information sources is a key methodological step for evidence synthesis, including scoping reviews. This study compares the performance of large language models with conventional systematic literature search methods in identifying published delirium clinical practice guidelines.

Methods

Comparative study using parallel searches. Four large language models (ChatGPT, Google Gemini, Claude and Grok) were queried using structured prompts, with results assessed for precision, sensitivity, and efficiency, versus a conventional systematic literature search using established bibliographic databases and grey literature sources.

Results

Large language models sourced 17 of the 45 guidelines identified through a comparable conventional search, missing 28, but also identified two guidelines not found through the conventional search. The sensitivity of individual large language models ranged from 9% to 29%, using the conventional search as a reference standard. Of the 19 guidelines retrieved through large language models, 63% were deemed high quality, compared with 56% identified through the conventional search. Precision of individual large language models varied from 22% to 60%, far exceeding the precision of the conventional search (1.7%). Large language model-based searches required substantially less execution time compared to conventional methods.

Conclusions

While large language models have a significantly lower yield in retrieving clinical guidelines compared to the conventional search, their efficiency, along with their capacity to identify some unique articles, suggests a possible role in complementing conventional search methods in scoping reviews.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12874-026-02941-x.

Introduction

Artificial intelligence (AI) systems have the capacity to transform academic research, and will likely revolutionise how researchers investigate, analyse, and produce knowledge [1]. AI systems are designed to operate with varying degrees of autonomy [2]. While AI encompasses a broad spectrum of technologies, some are of particular interest in academic research. Machine learning is a subset of AI that enables systems to learn directly from data [3]. Examples include machine learning classifiers that apply algorithms to categorise data [4], natural language processing (NLP) that enable machines to understand and generate human language [5], and deep learning, a machine learning subtype that uses neural networks to learn complex patterns [3]. Large language models (LLMs), a branch of generative AI, are a form of deep learning that came to the fore with the launch of ChatGPT in 2022. LLMs are trained on large datasets to generate new content in response to prompts [6].

While there is much excitement about AI’s potential, there are also many legitimate concerns about its use [7, 8]. Although shallow machine learning algorithms are interpretable by humans, most machine learning algorithms are untraceable [3]. Their ‘black box’ architecture results in a lack of transparency [6, 9]. AI models can inherit biases from training data, exhibit prompt bias, fabricate citations, and introduce both random and systematic errors [6, 8, 9]. These potential challenges pose a conflict with the research integrity principles of reliability, honesty, respect and accountability [9]. Four leading organisations on evidence synthesis have issued a joint statement on the responsible use of AI, noting the need for author accountability, human oversight, methodological integrity, and transparent reporting [10]. There is also an environmental impact of using AI given its energy demands [8, 11], which raises ethical concerns in the context of the global climate change crisis.

On the other hand, integrating AI tools into research offers advantages, particularly in improving efficiency [4, 10]. Evidence synthesis is the research process of collecting, analysing, and integrating findings from multiple studies to provide a comprehensive understanding of a research question or topic. It requires rigorous, transparent and reproducible methods [10]. Scoping reviews are a form of evidence synthesis with systematic steps including literature searching, screening, data charting and analysis [12–14]. This process is time-consuming, and with the volume of published literature increasing annually [15, 16], reviews risk becoming outdated by the time they are published. AI tools that automate review steps are appealing, and have been used to complete different steps in evidence synthesis processes, with varying degrees of success [10, 17–19,20]. One study found that as well as being faster, AI tools were more accurate than human-based methods in data extraction [21], while another demonstrated AI’s potential by completing a systematic review in under two weeks [22].

Different types of AI can be used throughout the evidence synthesis process. Machine learning classifiers trained on prior screening decisions can automate inclusion and exclusion decisions [23]. Rule-based algorithms can be used for literature retrieval, screening, data extraction and critical appraisal, and platforms including Covidence and Rayyan have integrated AI-assisted screening tools [4]. Commercial platforms including Elicit, Consensus and Perplexity use retriever-augmented generation to integrate LLMs with Semantic Scholar, enabling natural language queries to generate a list of relevant research papers [23].

Delirium is an acute and fluctuating neuropsychiatric syndrome characterised by disturbances in attention, awareness, and cognition [24]. Clinical practice guidelines provide recommendations for healthcare professionals that are informed by a systematic review of evidence [25]. Their goal is to improve healthcare quality, consistency, and cost-effectiveness [25–27]. This study explores the performance of LLMs in completing a literature search of delirium clinical practice guidelines. As a comparator, we used the results of a scoping review of delirium clinical practice guidelines [28]. The specific research question is:

How does the performance (sensitivity, precision, and efficiency) of LLMs compare with conventional literature search methods in identifying delirium clinical practice guidelines for a scoping review?

Methods

AI tools search strategy

Free versions of four common LLMs were selected, namely ChatGPT Version 4o (developed by Open AI), Gemini Version 2.5 Flash (developed by Google DeepMind), Claude Version Sonnet 4 (developed by Anthropic) and Grok 4 (developed by xAI). These LLMs were selected as they are publicly accessible and widely adopted, with freemium versions. AI tool use adhered to the PRISMA-trAIce checklist for documenting AI use in systematic literature reviews [29]. Search prompts were created by instructing each LLM to develop prompts based on both the MEDLINE Ovid and Google Scholar Boolean search strategies used in the conventional search strategy (Table 1). The Boolean search strategies were used as an initial prompt so that the AI search was anchored in the same conceptual framework as the conventional search strategy.These zero-shot prompts, which provide the task description without examples, were then enhanced through trial and error to gain more sensitive results (full details in Supplementary File 1). The prompt yielding the most results was then used across all four LLMs: “Search the web for current (since 2010) clinical practice guidelines on delirium, including synonyms like acute confusional state, acute brain failure, or toxic psychosis. Focus on documents labelled as clinical practice guideline, consensus statement, practice parameter, or best practice recommendation from reputable organizations or journals. Provide the most exhaustive list possible.” This combined four prompt engineering patterns: question refinement (e.g., scope, clarifying terms), context manager (e.g., timeframe, domain, sources), fact check list (e.g. criteria for correctness) and recipe pattern (structured, step-by-step approach) [31]. Searches were performed in November 2025, with articles published after August 2025 excluded to match the conventional search date range. One reviewer [MF] completed both screening stages. Results were managed using Excel (Microsoft 365 Version 2503).

Table 1.

Boolean operators used as input for developing LLM search prompts

‘Delirium’ Boolean search filter adapted from page 65 of NICE (2023) Delirium: prevention, diagnosis and management [A] Evidence review for diagnostic accuracy of tests to identify delirium [30]; modified as necessary to suit platform(s): https://www.nice.org.uk/guidance/cg103/evidence/a-diagnostic-accuracy-of-tests-to-identify-delirium-pdf-11320715486
‘Guideline’ Boolean search filter adapted from: Canada’s Drug Agency (2025) Guidelines - Standard; modified as necessary to suit platform(s): https://searchfilters.cda-amc.ca/
MEDLINE Ovid	Convert the following MEDLINE Ovid search into a search prompt for [INSERT LLM] to use to search for clinical practice guidelines on delirium: MEDLINE via Ovid 1 exp Delirium/ 2 Confusion/ 3 deliri*.tw. 4 (confus* adj3 state*).tw. 5 (acute* adj3 (confus* or “brain syndrome” or “brain failure”)).tw. 6 organic psychosyndrome.tw. 7 psychoorganic syndrome.tw. 8 psycho-organic syndrome.tw. 9 toxic confusion*.tw. 10 toxic psychosis.tw. 11 or/1–10 12 (guideline or practice guideline or consensus development conference or consensus development conference, NIH).pt. 13 (guideline* or standards or consensus* or recommendat*).ti. 14 (practice parameter* or position statement* or policy statement* or CPG or CPGs or best practice*).ti. 15 (care adj2 (path or paths or pathway or pathways or map or maps or plan or plans or standard)).ti. 16 ((critical or clinical or practice) adj2 (path or paths or pathway or pathways or protocol*)).ti. 17 (algorithm* and (pharmacotherap* or chemotherap* or chemotreatment* or therap* or treatment* or intervention*)).ti. 18 (algorithm* and (screening or examination or test or tested or testing or assessment* or diagnosis or diagnoses or diagnosed or diagnosing)).ti. 19 (guideline* or standards or consensus* or recommendat*).au. 20 (guideline* or standards or consensus* or recommendat*).ca. 21 systematic review.ti, pt, kf, sh. and (practice guideline* or treatment guideline* or clinical guideline* or guideline recommendation*).ti, ab, kf. 22 or/12–21 23 11 and 22
Google Scholar	Convert the following Google Scholar search into a search prompt for [INSERT LLM] to use to search for clinical practice guidelines on delirium (“delirium” OR “confusion” OR “delirious state” OR “acute confusion” OR “brain syndrome” OR “brain failure” OR “organic psychosyndrome” OR “psychoorganic syndrome” OR “psycho-organic syndrome” OR “toxic confusion” OR “toxic psychosis”) AND (“clinical guideline” OR “practice guideline” OR “consensus statement” OR “consensus development conference” OR “NIH consensus” OR “position statement” OR “policy statement” OR “CPG” OR “CPGs” OR “best practice” OR “care pathway” OR “clinical pathway” OR “care plan” OR “critical path” OR “treatment algorithm” OR “screening algorithm” OR “diagnostic algorithm”) Custom range 2010–2025

Conventional search strategy (reference set)

The conventional search drew on a NICE unvalidated search block for ‘delirium’ and the CDA validated ‘guideline’ filter (Table 1). This reflects a broader pattern, whereby search filters are more commonly developed for a study design or study focus than a specific health condition [32]. The search strategies were modified as necessary for the various databases and grey literature sources by a researcher [MF] and a UCC librarian [BH]. Four academic databases were searched: MEDLINE Ovid, Embase, Cumulative Index to Nursing and Allied Health Literature (CINAHL) and PsycInfo. Grey literature sources comprised a grey literature database (Turning Research into Practice, TRIP), Google (first 200 results), Google Scholar (first 200 results), three guideline repositories (ECRI Guidelines Trust, Guideline Central, Guidelines International Network (GIN)), and 13 organisation websites (Supplementary File 1). Reference lists of included articles were also screened. While the scoping review search also included guidelines sourced via expert knowledge, these were excluded to ensure a like-for-like comparison. Search terms were modified as necessary for the various data sources (Supplementary File 1). Searches were performed in August 2025 and covered guidelines published since 1st January 2010. Earlier versions of a guideline superseded by a later version were excluded. Two reviewers completed title and abstract screening, and subsequent full text screening, with conflicts resolved by a third senior reviewer [ST]. Rayyan [33], Zotero (7.0.7) [34] and Excel (Microsoft 365 Version 2503) were used to manage the results. Guidelines identified by this conventional search and screening process were considered the reference standard of relevant guidelines.

Definitions

A search filter is a set of search terms designed to identify study designs, topics or other issues. The performance of a search filter can be validated against a gold standard (also referred to as reference set) [35]. Sensitivity and precision are key metrics for assessing search filter performance in retrieving guidelines [36]. The ideal outcome is a search that maximises sensitivity while maintaining sufficient precision [36]. Sensitivity (or recall) is the number of relevant records retrieved by the search compared to the total number of relevant records in the reference set, while precision is the number of relevant records retrieved by the search compared to the total number of (relevant plus irrelevant) records retrieved [35]. For our study, the conventional search results act as a reference set. We define efficiency as the time required to develop the searches, execute the searches, and screen the results.

Results

Prompt iteration

Search prompts developed by the LLM’s initial conversion of the Boolean prompts were iterated to maximise the sensitivity of the AI tools’ output (Table 2). Details of search prompt inputs and outputs are provided in Supplementary File 1.

Table 2.

Search prompt iteration

Prompt description	No. of articles identified
	ChatGPT	Gemini	Claude	Grok
1. Prompt created by LLM based on MEDLINE Ovid Boolean search filter	7	5	6	3
2. Prompt created by LLM based on Google Scholar Boolean search filter	10	4	13	3
3. Prompt iterations and question refinement (e.g. adding request to expand list of LLM results)	3 A = 12 3B-=20	3 A = n/a 3B = 14	24	13
4. Final prompt iteration ChatGPT search prompt that generated 20 results + “provide the most exhaustive list possible”	26	15	40	18

Guideline retrieval

A total of 99 articles were identified, by Claude (n = 40), ChatGPT (n = 26), Grok (n = 18) and Gemini (n = 15). While duplicates within individual LLM results were removed, duplicates common across LLMs were not removed, to show the relative effectiveness of each LLM in identifying relevant articles. Record exclusion was completed by the reviewer, not AI. One article published after the timeframe of the conventional search was excluded. A search summary table [37] presents the retrieval results by source (Table 3). The highest number of relevant guidelines was identified by Claude (n = 14) [38–51] and ChatGPT (n = 12) [38–42, 44, 46–48, 50, 52, 53], with fewer relevant guidelines sourced by Gemini (n = 9) [38, 42, 44, 46, 48, 49, 54, 55] and Grok (n = 4) [39, 42, 54, 55]. Combined, the LLMs identified 19 unique guidelines. Two of these were found by hand searching included LLM article reference lists [56, 57] and both were identified in one guideline [38] which appeared in both academic and grey literature searches. Both ChatGPT and Claude identified unique articles not captured by the conventional search [51, 53]. See Fig. 1 and Table 3. The LLMs failed to find 28 guidelines sourced through the conventional search [58–85].

Table 3.

Search summary table: mapping guidelines to source(s). The two guidelines uniquely identified by LLMs are shown in red

Fig. 1.

PRISMA flow chart comparing guideline identification using conventional search versus guideline identification using LLMs

Guideline quality

The Appraisal of Guidelines for Research and Evaluation Instrument (AGREE II) is designed to assess the methodological quality of clinical practice guidelines [86]. Of the 19 guidelines identified by LLMs combined, 63% (n = 12) were high-quality, defined as an overall assessment score ≥ 4 (max. 7) and a ‘rigour of development’ score > 60%. Although the conventional search identified more high-quality guidelines (n = 25) they represented a smaller proportion (56%) of the overall number retrieved. See Table 3.

Sensitivity

Sensitivity, defined as the number of relevant guidelines retrieved (true positives) compared to the total number in the reference set, was low. Taking the conventional search as the reference standard, the sensitivity of individual LLMs ranged from 9% to 29%, with Claude correctly identifying the highest number of relevant guidelines (Table 4). Combined, their sensitivity was 38%.

Table 4.

LLM performance: speed, precision and sensitivity

	Time to build search string / prompt	Time taken to run search	Time for (one reviewer) title and abstract screening‡	Time for (one reviewer) full text screening‡	No. results NR	True positives TP	TPs that are high quality guidelines (% )	False positives FP	False negatives FN	Precision	Sensitivity§
Conventional search	5 h*	3 h†	36 h	7 h	2,676	45	25/45 (56%)	2,631	n/a	1.7%	n/a
ChatGPT	1 h	< 1 min	15 min	0.8 h	26	11 (plus 1 ‘new’)	9/11 (82%)	14	34	42%	24%
Gemini		< 1 min	10 min	0.6 h	15	9	7/9 (78%)	6	36	60%	20%
Claude		2 min	25 min	1 h	40	13 (plus 1 ‘new’)	10/13 (77%)	25	32	33%	29%
Grok		< 1 min	10 min	0.3 h	18	4	3/4 (75%)	13	41	22%	9%
All 4 LLMs combined	1 h	< 5 min	1 h	2.7 h	99	17 (plus 2 ‘new’)	13/17 (76%)	79	28	17%	38%

The no. of guidelines identified in conventional search is used as the reference standard (i.e., 45 guidelines)True positive (TP) = no. guidelines correctly identified (i.e. also identified in conventional search); excludes ‘new’ guidelines identified by LLMsFalse positive (FP) = no. guidelines incorrectly identified as eligible. One article correctly identified as a guideline by both Claude and Grok but published post-Aug 2025 has not been recorded as a false positive

False negative (FN) = no. guidelines incorrectly excluded (included by conventional search but missed by LLMs) = 45-TP

Precision = no. of relevant guidelines retrieved compared to total no. of all records retrieved = TP/(TP + FP) = TP/NR

Sensitivity = no. of relevant guidelines retrieved compared to total no. in the reference set = TP/45

* Estimate, based on meeting with librarian, modifying search strings for four different academic databases and one grey literature database

† Estimate, based on time taken to run search on 23 different sources: five databases (including four academic databases, each with 24–26 search terms), Google, Google Scholar, three guideline repositories and 13 organisation websites. Does not include time taken to save search results

‡ Based on estimated title and abstract screening speed of 60 articles/hour, and full text screening speed of 15 articles/hour

§ Sensitivity is based on TP which in some cases includes unique articles not identified by the reference standard

Precision

Precision, defined as the number of relevant guidelines retrieved (true positives) compared to the total number of records retrieved, varied between 22% and 60% depending on the LLM, with Gemini delivering the best performance (Table 4). All LLMs exceeded the precision of the conventional search (1.7%) which required extensive screening due to high false-positive rates.

Efficiency

Efficiency was measured by assessing the time to develop and execute the searches, and screen the results. Developing the LLM search prompt required less time than constructing the conventional search filters (Table 4). The latter involved adapting search strings with over 20 search statements for four different academic databases, whereas the Boolean search filters were leveraged as input for the LLMs with subsequent iteration of a final prompt through trial and error (Supplementary File 1). The AI tools were quicker at running the searches, taking less than five minutes. In contrast, the conventional approach required an estimated four hours to search across 23 separate sources. The reduced volume of results for title and abstract screening for LLMs (n = 98) compared to the conventional search (n = 2,147) resulted in an estimated time saving of 35 h per reviewer. Similarly, the time for full text screening per reviewer decreased from seven hours to less than three hours due to the reduced number of articles.

Discussion

This study compared the performance of LLMs with conventional systematic literature search methods in identifying published delirium clinical practice guidelines.

Relative to our conventional search, the low sensitivity of LLMs in identifying relevant guidelines aligns with other research demonstrating the limited utility of AI tools in retrieving relevant studies for evidence synthesis [17, 19]. The LLMs sensitivity may even be inflated for this particular topic, given that grey literature is known as an important source for identifying delirium clinical practice guidelines [87] and thus more likely to be identified by LLMs. A different research question, where evidence is largely restricted to paywalled articles and subscription-based databases and thus not publicly available, may yield even lower sensitivity. Our finding that results were highly dependent on prompt quality is consistent with other studies that emphasise the critical role of prompt design in shaping LLM outcomes [20, 88].

LLMs are designed to interpret and generate text rather than to perform search tasks. Applying them to a literature review search may not be the best use of their capability. This is evidenced by the fact that the conventional approach, which included a search of the first 200 results of Google, identified 17 guidelines (Table 3), exceeding the highest yield of a single LLM (n = 14). AI tools may be more beneficial for other evidence synthesis steps [18, 19, 89, 90]. A scoping review of systematic review tasks found AI tools for literature search yielded mostly neutral or unpromising results, whereas AI tools applied to screening and data extraction were more promising [90] LLMs are also recognised as having a potential role in finding ‘seed references’ and keyword synonyms in the search methodology process [91].

Despite their limitations in identifying all published delirium clinical practice guidelines, the LLMs demonstrated some advantages compared to the conventional search. Firstly, while the low sensitivity results highlight the superiority of an extensive conventional search, LLMs also identified two unique articles that were missed by the conventional. This suggests a role for LLMs as an additional quality measure in a search. Secondly, although the LLM search benefited from time savings by leveraging the Boolean strings generated for the conventional search to create its initial prompts, the time taken to execute these search prompts in the LLMs was significantly faster. However, if LLM use were to become an additional search source, it would become an additional time resource demand. Thirdly, the LLM’s high precision compared to the conventional search reduced the volume of irrelevant results, thereby improving efficiency. Fourthly, the LLMs were marginally more likely to identify high quality guidelines, although the reason for this is not clear. It may be a coincidence or reflect how clinical practice guidelines are disseminated rather than indicating LLMs are better at identifying higher quality guidelines. These benefits suggest that while low sensitivity rules out LLMs as a replacement for a conventional search, AI tools may have potential as a complement to conventional search methods, a finding supported by other research [92, 93]. A second use might be as a pre-screen before proceeding to a full review, where the LLM results would give an indication of the full field.

While it may be possible to outsource some of the workload to AI tools, it is not possible to outsource research responsibility. There is consensus that human oversight remains crucial in AI utilisation [18, 89, 90, 94] As evidence synthesis is built on the core principles of rigour, transparency and reproducibility, the integration of AI and automation should comply with experts’ recommendations on using it responsibly [4]. Generative AI deployment raises legitimate concerns for research, particularly regarding bias, hallucinations, transparency and reproducibility [9]. Researchers remain accountable and responsible not only for their research outputs, but also in what they share with AI tools, given that generative AI may be trained on uploaded data [9].

As AI tools continue to evolve and given their potential to improve over time, researchers should remain open to considering their use, while being cognisant of the challenges and concerns around their deployment. Whether or not LLMs or other AI tools will fully replace conventional literature search approaches is uncertain, but their potential as a complementary tool merits consideration. As the integration of AI into academic research is likely to continue, researchers must develop the necessary competencies to use them in an effective, responsible, legal and ethical manner.

This study has limitations. Firstly, as there are over 70 different AI tools for literature search [95], we did not systematically assess all AI tools that could have been used, instead selecting a pragmatic shortlist of four common LLMs. Other LLMs, paid versions of the selected LLMs, or other AI tools such as Consensus, Elicit and Perplexity, may have yielded better results. Secondly, although efforts were made to iterate and improve the prompts, no specialist expertise was used to guide prompt engineering, which could impact the results. Thirdly, while the conventional search screening was completed by two reviewers, LLM screening was completed by one reviewer. Fourthly, the selection of MEDLINE Ovid and Google Scholar Boolean prompts over others (such as Embase) was arbitrary. A further limitation concerns the final LLM prompt, refined through iterative trial and error but grounded in the initial Boolean search strategies. The expertise and mindset required to construct those searches likely contributed to the prompt’s quality and subsequent results. Finally, the LLM performance was compared against a human-generated reference set, which itself was not validated.

Conclusions

The LLMs selected for this study demonstrated significantly lower sensitivity compared to conventional literature search methods in identifying delirium clinical practice guidelines. However, the speed, precision, and ability to identify some unique guidelines suggests generative AI may be a useful complementary resource in systematic literature searches, as an augmentation or pre-screening step. Future research should continue to evaluate the effectiveness and efficiency of AI tools in literature searching, as their performance may improve with new version releases. At the same time, researchers must remain aware of the inherent risks in using AI and their professional responsibility for the integrity of research outputs.

Abbreviations

AGREE II

Appraisal of Guidelines for Research and Evaluation Instrument

AI

Artificial Intelligence

CDA

Canada's Drug Agency

CINAHL

Cumulative Index to Nursing and Allied Health Literature

FP

False Positive

GIN

Guidelines International Network

LLM

Large Language Model

NICE

National Institute for Health and Care Excellence

NLP

Natural Language Processing

NR

Number of Results

PRISMA

Preferred Reporting Items for Systematic reviews and Meta-Analyses

TP

True Positive

TN

True Negative

TRIP

Turning Research into Practice

Authors’ contributions

MF: Conceptualization, Data Curation, Formal Analysis, Methodology, Project Administration, Original Draft Preparation, Writing – Review & Editing. RM: Methodology, Writing – Review & Editing. BH: Methodology, Writing – Review & Editing. SB Methodology, Writing – Review & Editing. AML: Conceptualization, Writing – Review & Editing. ST: Conceptualization, Project Administration, Supervision, Writing – Review & Editing. All authors read and approved the final manuscript.

Funding

No funding.

Data availability

Supplementary File 1 includes details on the conventional search databases and search filters used as the comparator as well as details on the LLM prompt iterations and outputs. The study protocol outlining the conventional search used as the reference standard for this article is available on the HRB website [28].

Declarations

Ethics approval and consent to participate

As this article does not contain any studies with human or animal participants, informed consent and ethical approval were not required.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.