Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies

ABSTRACT

Background

Elicit AI aims to simplify and accelerate the systematic review process without compromising accuracy. However, research on Elicit's performance is limited.

Objectives

To determine whether Elicit AI is a viable tool for systematic literature searches and title/abstract screening stages.

Methods

We compared the included studies in four evidence syntheses to those identified using the subscription‐based version of Elicit Pro in Review mode. We calculated sensitivity, precision and observed patterns in the performance of Elicit.

Results

The sensitivity of Elicit was poor, averaging 39.5% (25.5–69.2%) compared to 94.5% (91.1–98.0%) in the original reviews. However, Elicit identified some included studies not identified by the original searches and had an average of 41.8% precision (35.6–46.2%) which was higher than the 7.55% average of the original reviews (0.65–14.7%).

Discussion

At the time of this evaluation, Elicit did not search with high enough sensitivity to replace traditional literature searching. However, the high precision of searching in Elicit could prove useful for preliminary searches, and the unique studies identified mean that Elicit can be used by researchers as a useful adjunct.

Conclusion

Whilst Elicit searches are currently not sensitive enough to replace traditional searching, Elicit is continually improving, and further evaluations should be undertaken as new developments take place.

Summary

• AI tools, such as Elicit, have been developed to improve the efficiency of evidence synthesis processes, including the identification of studies.

• Using four case study evidence syntheses Elicit searches had a sensitivity between 25.5% and 69.2% (39.5% average) and precision between 35.6% and 46.2% (41.8% average).

• Elicit identified some unique studies that met the inclusion criteria for each of the case study evidence syntheses.

• Based on our results, Elicit is recommended as a supplementary search tool, but not sufficient to replace traditional search methods

• Elicit is constantly improving and developing its systems, thus independent researchers should continue to evaluate its performance.

1. Background

Systematic reviews aim to answer a clear question by collating and synthesizing existing research on a specific topic [1, 2]. Searching on multiple databases and sources, identifying, selecting, and appraising studies are all part of the traditional methods of systematic reviews and other evidence syntheses, one that relies on rigorous, time‐consuming, and manual processes, often undertaken collaboratively as part of a team [1, 3, 4]. Elicit AI aims to simplify and speed up this process by up to 80%, with a claimed time saved per review of 16 h [5]. Elicit AI is an AI‐powered research assistant originally developed by Ought, a nonprofit machine learning (ML) research lab based in the US [6]. Since then, Elicit has become a for‐profit company backed by venture capital investment [7]. Elicit allows the user to ask a research question and from there, is able to significantly automate several of the key stages in the systematic review process by searching over 126 million papers, generating screening criteria, undertaking data extraction, and providing a first draft of the review in a very short period of time [8]. Traditionally, systematic reviews can take from 6 to 18 months [9]. Policy makers, like NICE, structure guidelines based on systematic reviews so there is a trade‐off between quality and time [10]. Waiting for a systematic review to be completed may have real world implications on patients who are being treated using potentially out‐of‐date guidelines. Alongside the speed and ease of use, Elicit claims to allow for more evidence to be considered and to avoid human bias without compromising on accuracy [11].

AI is projected to contribute $15.7 trillion to the global economy by 2030 [12] and in the UK alone, the number of AI companies has increased by 600% over the last 10 years [13] (Artificial Intelligence [14], As the use of AI in scientific research increases, the impact on research integrity becomes an ever more important factor to consider. Between 2003 and 2024, there has been an increase in over 15 000 papers published in Pubmed with the term ‘artificial intelligence’ [15]. Of these, only 3 included both the terms ‘artificial intelligence’ and ‘research integrity’. This highlights that although research integrity is a crucial consideration when using AI tools like Elicit, there is a lack of literature surrounding this topic. Elicit may be able to significantly speed up the systematic review process, but at what cost to the rigour and reliability of traditional methods. Recent evidence reinforces this concern. Clark et al. [16] conducted a systematic review of generative AI tools in evidence synthesis, and concluded that they generally are “not yet ready to be used without human oversight.” Likewise, Lieberum et al. [17] carried out a scoping review in 2024 on large language model workflows and reached the same verdict. However, crucially neither review evaluated Elicit ‐ instead mainly focussing on ChatGPT ‐ leaving a clear gap that our present study aims to address.

After compiling an answerable question, the next part of the systematic review process is to create comprehensive searches of multiple databases and other sources. This step is key to the quality of any systematic review. It is this stage of the review process that the present study evaluates by assessing the value of Elicit's literature search facility.

Our objective is to compare the use of Elicit in identifying the included studies in evidence syntheses reviews compared to traditional search methods in evidence syntheses.

2. Methods

We selected four different evidence syntheses (two systematic reviews, one umbrella review and one scoping review) in three different areas: public health (two reviews), pharmacology, and surgical procedure. This allows a comparison of Elicit across a range of subject areas to see how Elicit performs in each, thus improving the generalizability of our results. Two of the evidence syntheses (one umbrella review and one scoping review) were conducted in 2024 by one of the authors with the searches undertaken by an experienced information specialist. The other two were recent systematic reviews and were selected from the Cochrane Database of Systematic Reviews as Cochrane Reviews tend to be of higher quality than those published in peer review journals [18, 19, 20, 21, 22, 23]. The case studies were selected from very recently published evidence syntheses, ensuring they did not overlap with the GPT‐3 training period and thereby avoiding contamination.

Elicit uses GPT‐3 as its underlying large language model (LLM) and searches the Semantic Scholar database to find publications [6], searching over 126 million papers. Semantic Scholar is an open access AI‐based search engine providing ranked citations. Elicit has the advantage of helping to formulate the research question and screening articles identified against prespecified inclusion criteria as well as features to assist in the other key processes in systematic reviews.

For this evaluation, we used the Review feature in the subscription‐based version Elicit Pro. Elicit is available in three tiers: Basic (free), Plus, and Pro. The Basic version is intended for students and casual exploration and includes access to core Elicit functions such as literature search and basic extraction. Both Plus and Pro are subscription‐based, with the Plus tier offering increased usage limits. However, the Review feature is only available in Elicit Pro, which provides the highest usage limits and is the most comprehensive subscription‐based tier of Elicit. Elicit also offers three different modes tailored to various research tasks: Find Papers mode for literature searching, Research Report mode for generating summaries from selected papers, and Review mode, which we chose because it is the dedicated workflow specifically designed to support systematic reviews.

We translated the original research question from each evidence synthesis based on its PICO elements into an Elicit query. Next, Elicit finds the 500 most relevant studies based on the query, and the title and abstract. After this, Elicit generates automated screening criteria for these 500 studies. Some of the initial criteria did not align with those in the review, so Elicit makes it easy for us to adapt and specify the screening criteria ‐ this does not change the 500 potentially eligible studies. We manually adjusted the screening criteria where necessary to match the PICO elements and inclusion criteria of the original comparative reviews. This ensured that the criteria were aligned before Elicit carrying out the screening step. Once Elicit identified the eligible studies, we exported all 500 studies into Microsoft Excel and compared the studies found in Elicit to those included in the review. Elicit makes the screening process very transparent as it provides a spreadsheet with individual columns for every inclusion criterion and if each of the studies meets it. Finally, Elicit generates a score based on the number of criteria met by the study, factoring in the importance of each one. This score is then used to determine the overall outcome of the study, with the threshold score for inclusion set automatically by Elicit. Elicit stresses the fact that although it can fully automate the search, at any point in time, the user is able to ‘manually override any disagreements.’ [24]. We noted three categories: studies that Elicit did not find, studies that Elicit found but excluded based on not meeting the screening criteria, and studies that Elicit found and included. We emailed the authors of the original reviews with a list of studies identified by Elicit that hadn't been included in the original review. With the help of the authors, we were able to assess whether any of these studies met the original inclusion criteria–representing unique additional findings. We calculated the sensitivity and precision of each search using the following formulae: Sensitivity—Number of included records retrieved/Total number of included records *100, Precision—Number of included records retrieved/Total number of records retrieved *100 [25].

2.1. Case Study 1: Vaping Harms

This was an umbrella review on the acute and long‐term harms of vaping in young people under the age of 25 [26]. The authors of the review searched the KSR Evidence database on OVID, with no language restrictions on the search strategy. They also conducted a separate search for umbrella reviews on MEDLINE, Embase, and PsycINFO as these types of review are not indexed in the KSR Evidence database. The search strategy was limited to a publication date of 2015 onwards, reflecting the shift in the type of e‐cigarette device being used by young people at that time. The search strategy was last conducted on the 21st November 2024. In total, 381 records were screened at title and abstract stage with 56 reviews included in the final umbrella review. Thus, the precision of the searches was 56/381, 14.7%, with a Number Need to Read of 7.

For Elicit, the question asked was “What systematic reviews exist that comprehensively document the short‐term and long‐term health risks associated with vaping in populations aged 10‐24 years?” We asked Elicit the question on the 24th February 2025.

2.2. Case Study 2: Physical Activity

This was a scoping review mapping the evidence from economic analyses on the cost‐effectiveness of population‐based interventions that could be funded or provided by local authorities in the UK to increase physical activity and to inform decision making at local and national level [27]. The search was focused on economic literature from 2015 onwards, and contained two main segments: physical activity public health interventions in the UK and the study design being economic analyses. Fifteen databases were searched in this review (MEDLINE, Embase, CEA Registry, EconLit, Research papers in economics (RePEc) ‐IDEAS), Social Policy and Practice, Healthcare Management Information Consortium (HMIC), Social Care Online (up to 2022), Social Systems Evidence, ASSIA, SPORTDiscus, BNI, PsycINFO, DoPHER, TRoPHI) as well as citation searching and browsing of key websites and organisations. In total 4868 records were screened at title and abstract with 50 studies included in the final scoping review. Thus, the precision of the searches was 50/4868, 1.03%, with a Number Need to Read of 97.

For Elicit, the question asked was “What is the cost‐effectiveness of population‐based interventions that could be funded or provided by local authorities in the UK to increase physical activity, based on economic analyses published since 2015.” We asked Elicit the question on the 11th March 2025.

2.3. Case Study 3: Fertility Treatment

This was a systematic review published in the Cochrane Library evaluating the effectiveness and safety of vasodilators in women undergoing fertility treatment [28]. The inclusion criteria consisted of: population ‐ women of any age undergoing fertility treatment, intervention ‐ vasodilators administered via any route with or without other agents, comparator placebo/no treatment/other active intervention, and the primary outcomes being live birth or ongoing pregnancy and the side effects of vasodilators. The following electronic databases, trial registers, and websites were searched in this review: the Cochrane Gynaecology and Fertility Group (CGF) Specialised Register of controlled trials, the Cochrane Central Register of of Controlled Trials, via the Cochrane Register of Studies Online (CRSO), MEDLINE, Embase, PsycINFO, the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Knowledge, the Open System for Information on Grey Literature in Europe (OpenSIGLE), the Latin American and Caribbean Health Science Information Database (LILACS), clinical trial registries, and the reference lists of relevant articles. In total, 347 records were screened at title and abstract stage with 48 publications representing 45 studies in the final systematic review. Thus, the precision of the searches was 48/347, 13.8%, with a Number Need to Read of 7.

For Elicit, the question asked was “Evaluate the effectiveness and safety of vasodilators in women undergoing fertility treatment. Outcomes including (endometrial thickness, adverse drug reactions, live births, multiple pregnancy, ectopic pregnancy, clinical pregnancy, and miscarriage.” We asked Elicit the question on the 7th March 2025.

2.4. Case Study 4: Breast Reconstruction

This was a systematic review published in the Cochrane Library assessing ‘the effects of implants vs autologous tissue flaps for postmastectomy breast reconstruction on women's quality of life, satisfaction, and short‐ and long‐term surgical complications’ [29]. The inclusion criteria consisted of women undergoing primary breast reconstruction after mastectomy for breast cancer treatment or risk reduction, implant‐based reconstruction compared to any autologous‐tissue reconstruction. With the primary outcomes being patient‐reported outcomes (BREAST‐Q, BRECON‐31, and EORTC QLQ BRECON‐23), short‐ and long‐term complications or oncological outcomes. The Cochrane Breast Cancer Group's Specialised Register, CENTRAL, MEDLINE, Embase, and two trials registries were searched for included studies in this review. In total, 6308 records were screened at title and abstract stage with 41 publications representing 35 studies in the final systematic review. Thus, the precision of the searches was 41/6308, 0.65%, with a Number Need to Read of 154.

For Elicit, the question asked was “Assess the effects of implants vs. autologous tissue flaps for post mastectomy breast reconstruction on women's quality of life, satisfaction and short‐ and long‐term surgical complications.” We asked Elicit the question on the 6th March 2025.

3. Results

3.1. Case Study 1: Vaping Harms

Elicit identified 38 reviews that met the screening criteria according to Elicit. Of the 38, 14 were already included in the original umbrella review. 22 studies identified by Elicit did not meet the inclusion criteria for the umbrella review. One was a scoping review [30], four were protocols of reviews [31, 32, 33, 34], four were not systematic reviews Weni Nur [35, 36, 37, 38], six weren't looking at the harms [39, 40, 41, 42, 43, 44], three had no analysis of young people Arya Marganda [45, 46, 47], one was a pre‐print [48] of a study that was already included in the umbrella review [49] and three weren't available in English language [50, 51, 52]. A further two studies [53, 54] that were identified by Elicit met the inclusion criteria of the umbrella review and would have been included if they had been identified. Both reviews were not indexed on the KSR Evidence database at the time of searching; one was a report, and the other was not indexed on MEDLINE, Embase or PsycINFO.

Whilst Elicit included 14 of the originally included reviews, 42 of the reviews included by Golder were not included. 31 were not identified, and a further 11 were excluded based on multiple reasons such as ‘failure to meet health outcomes requirement’ [55, 56] and ‘uncertainty regarding specific age range.’ [57, 58, 59, 60]. Of the 31 unidentified included studies, 26/31 (83.9%) were indexed in Semantic Scholar, and 18/26 (69.2%) of these were open access (Figures 1, 2, 3, 4, 5, 6).

Figure 1.

Included Studies in Case Study 1.

Figure 2.

Included Studies in Case Study 2.

Figure 3.

Included Studies in Case Study 3.

Figure 4.

Included Studies in Case Study 4.

Figure 5.

Sensitivity Comparison of Elicit and Traditional Review for Each Case Study.

Figure 6.

Precision Comparison of Elicit and Traditional Review for Each Case Study.

The search in Elicit had a sensitivity of 27.6% (16/58) and a precision of 42.1% (16/38).

3.2. Case Study 2: Physical Activity

Elicit identified 30 reviews that met the screening criteria according to Elicit. Of the 30, 12 were also included in the original scoping review [26]. However, nine reviews considered eligible by Elicit were published before 2015 even though there was explicit screening criteria of a post 2015 publication date. The other eight studies identified by Elicit did not meet the inclusion criteria. Four were protocols [61, 62, 63, 64], two were not an evaluation of an intervention [65, 66], one did not include separate outcome data for physical activity [67] and one had no specific intervention evaluated in terms of Return on Investment [68]. Additionally, Elicit found one review [69] that would have been included in the original review had it been identified by the traditional searches. This study was published in the Journal of Transport and Health which was not indexed in the majority of the databases searched such as MEDLINE or Embase, however, it was indexed in PsycINFO but was not identified using the economic search filter used (although it was identified by the other facets of the search).

Elicit did not include 38 of the reviews included by Golder – 27 were not identified and a further 11 were excluded based on multiple reasons such as ‘limited by the individual level approach’ [70, 71, 72], ‘narrow employee population and workplace setting limit’ [73], ‘uncertainties around explicit UK context’ [74, 75, 76, 77] and ‘significant uncertainty regarding publication year’ [78, 79, 80]. Of the 27 unidentified included studies, 22/27 (81.5%) were indexed in Semantic Scholar, and 22/22 (100%) of these were open access.

The search in Elicit had a sensitivity of 25.5% (13/51) and a precision of 43.3% (13/30).

3.3. Case Study 3: Fertility Treatment

Elicit identified 78 studies that met the screening criteria according to Elicit. Of the 78, 32 were included in the original Cochrane Review. Similarly, to case study 1 and 2, Elicit identified four additional studies that would have been included had they been identified at the time of searching [81, 82, 83, 84]. Notably, all four RCTs found by Elicit alone were not indexed in either MEDLINE or Embase. The rest of the studies did not meet the inclusion criteria due to multiple reasons such as not being an RCT [85, 86].

Elicit did not include 16 of the RCTs included in the original Cochrane Review– 12 were not identified and a further four were excluded for multiple reasons such as ‘significant uncertainty exists regarding specific outcomes reported and the precise study design’ [87], ‘ doesn't specifically breakdown types of pregnancy.’ [88], ‘ambiguity about whether this fully aligns with the purposes of fertility treatment.’ [89] and ‘doesn't explicitly report endometrial thickness, live births, adverse reactions, ectopic pregnancy, miscarriage or multiple pregnancy.’ [90]. Of the 12 unidentified included studies, 7/12 (58.3%) were indexed in Semantic Scholar, and 4/7 (57.1%) of these were open access.

The search in Elicit had a sensitivity of 69.2% (36/52) and a precision of 46.2% (36/78).

3.4. Case Study 4: Breast Reconstruction

Elicit identified 45 studies that met the screening criteria according to Elicit. Of the 45, 12 were also included in the original Cochrane review. Additionally, Elicit identified a further four studies that weren't included in the original review, but met the inclusion criteria and in hindsight, would've been included. The following four are [91, 92, 93, 94]:.

Elicit did not include 29 of the studies included in the Cochrane review – 21 were not identified and a further eight were excluded for multiple reasons such as ‘absence of patient‐centred outcomes’ [95], ‘ minor uncertainty regarding direct comparison between reconstruction techniques’ [96], ‘ primarily focusing on costs and technical complications’ [97], ‘ cross‐sectional study’ [98], ‘quality of life measures and patient satisfaction were not measured’ [99], ‘ uncertainty regarding primary vs secondary reconstruction and patient reported measures’ [100] and ‘some population details remain implicit.’ [101] However [102], ‐ one of the eight excluded studies ‐ met all the inclusion criteria but wasn't included because the Elicit score was not high enough (it scored 4.8 with the cut‐off set at greater than 4.8). Of the 21 unidentified included studies, 21/21 (100%) were indexed in Semantic Scholar, and 7/21 (33.3%) of these were open access.

The search in Elicit had a sensitivity of 35.6% (16/45) and a precision of 35.6% (16/45).

Across all case studies, we observed that the stage of selecting the 500 most relevant studies via AI is neither as transparent or reproducible as using a traditional database search strategy such as in OVID Medline (Table 1).

Table 1.

Studies identified by Elicit and review with traditional searching methods for each case study.

	Elicit inclusion	Included in traditional review	Not included in traditional review and did not meet inclusion criteria	Not included in traditional review but did meet inclusion criteria	Elicit precision (%)	Elicit sensitivity (%)	Traditional searching Precision (%)	Traditional searching sensitivity (%)
Case Study 1: Vaping harms	38	14	22	2	16/38 (42.1%)	16/58 (27.6%)	56/381 (14.7%)	56/58 (96.6%)
Case Study 2: Physical Activity	30	12	17 (9 pre‐2015)	1	13/30 (43.3%)	13/51 (25.5%)	50/4868 (1.03%)	50/51 (98.0%)
Case Study 3: Fertility Treatment	78	32	42	4	36/78 (46.2%)	36/52 (69.2%)	48/347 (13.8%)	48/52 (92.3%)
Case Study 4: Breast Reconstruction	45	12	33	4	16/45 (35.6%)	16/45 (35.6%)	41/6308 (0.65%)	41/45(91.1%)

4. Discussion

The primary aim of this study was to assess Elicit's ability to identify studies that meet the inclusion criteria for an evidence synthesis. We hypothesized that Elicit may be able to assist in the literature search and screening of studies as part of the evidence synthesis process, and with improvements even replace traditional searching methods. We also sought to determine whether Elicit is comparable to traditional searching in terms of recall and precision.

Across all four case studies, Elicit was able to find and include additional studies that the original authors didn't. This may be due to Elicit's ability to search on a wider set of records [11], as it uses Semantic Scholar which indexes over 126 million articles. This, coupled with Elicit's speed and high precision makes it useful for conducting preliminary searches for example, costing a grant proposal, providing seed papers for improving traditional searches or testing search strategies, or determining whether there is a risk of an empty review. The fact that Elicit provides detail for each of the studies and reasoning for inclusion/exclusion in the screening process is an advantage over many other AI tools and it also allows the user to understand to some extent what Elicit is doing in real time and make adjustments ‐ strengthening the research integrity [15]. Whilst Elicit may not be at the stage of development to replace traditional searching methods, Elicit's ability to assist in the search process is very apparent through its speed and precision. Precision using Elicit across the four evidence syntheses had an average of 41.8% which is impressively high, compared to the average of the original reviews which was 7.55%. With such high precision, Elicit would be invaluable in the costing and proposal stage of a review. In addition to its potential role in identifying seed articles, Elicit could also serve as a useful tool for supplementary searches once traditional searching and screening have been completed.

Little independent research on the performance of Elicit has been conducted to date. Fenske and Otts [103] conducted a descriptive study comparing Elicit to PubMed and CINAHL. They surveyed 323 graduate nursing students, with the primary outcome being which resource they preferred and their opinions on Elicit. As the study was conducted in the fall of 2023, they used a beta version of Elicit. Of the 26% of students who preferred Elicit, 38.8% listed the user‐friendliness as a top strength. This aligns with our study as Elicit has a very shallow learning curve, which allows for it to be easily implemented as an auxiliary tool alongside a traditional literature search. By using Elicit as an adjunct, as shown in our case studies, it may allow the researcher to find additional studies that otherwise wouldn't have been identified.

However, in our study we noticed some key flaws in Elicit. For case study 2, Elicit deemed nine studies‐which were published before 2015‐ eligible even though we explicitly added to the screening criteria to only include articles published since 2015. This shows that even through manually adjusting screening criteria, Elicit is not perfect and studies can slip through the cracks in the screening stage. This potentially contradicts the benefit of being able to change the criteria by hand. Raising the question of the value of being able to make adjustments if Elicit doesn't always apply the criteria. The sensitivity for systematic review searches should reach at least 90% [104]. However, due to Elicit's limited sensitivity ‐ average of 39.5% ‐ this means that it currently cannot be used as a one stop shop in systematic review searches or other comprehensive reviews. Across all four case studies, an average of 80.9% of the unidentified included studies were indexed in Semantic Scholar, with 64.9% of these being open access. This suggests that the limited sensitivity observed is more likely due to Elicit's searching skills rather than the lack of access to the same journals. Case study 3 [28] had by far the highest sensitivity of 69.2%. A potential reason for this may be because the PICO structure and inclusion criteria for pharmacology reviews are more straightforward, well‐defined and binary. Alternatively, the two public health reviews had the lowest sensitivity, highlighting potential limitations with Elicit when the PICO is more ambiguous and complex.

Interestingly, in case study 2, Elicit retrieved a pair of true duplicate studies [105] – with the same DOI and the same journal publication—but treated them as distinct studies rather than recognising them as duplicates. Hence, they were scored individually, and they received two different scores: 4.1 (excluded) and 4.8 (included). This highlights possible inconsistencies with the screening process and questions whether Elicit is reliable in de‐duplication when the study appears in different sources.

Furthermore, in case study 4, the Elicit score threshold (as mentioned previously) was set at greater than 4.8, with [102] scoring 4.8, thus being excluded. However, several other studies—for example [106, 107, 108] ‐ also had a score of 4.8 but were included by Elicit. This highlights potential inconsistencies in Elicit's screening, proving that human oversight remains an integral part when interpreting borderline cases. Further investigation into the scoring process may explain why studies with identical scores receive different outcomes and whether supplementary criteria are affecting inclusion.

A recently published study by Bernard et al. [109] evaluated the repeatability, reliability, and accuracy of Elicit in one case study. The authors searched Elicit on the 18th of April 2023 and had similar findings to our study. Elicit identified three unique studies that weren't included in the umbrella review but did meet the inclusion criteria. The ability of Elicit to find studies that otherwise wouldn't have been included is a strong argument for Elicit as an adjunct. However, just like our study, Bernard et al. found that the sensitivity of Elicit is poor. As it was only able to find 3 out of 17 studies (17.6%) included in the umbrella review. In our case studies using the same methods of calculation as Bernard et al., Elicit found 25.0%, 24.0%, 66.7% & 29.3% respectively. Bernard et al. conducted their evaluation 2 years ago (April 2023). The higher sensitivities observed in our study may potentially reflect improvements to Elicit over time, although it is still unable to replace a traditional literature search.

Building on from both our study and Bernard et al., Tomczyk et al. [110] evaluated the search ability of Elicit against those of experienced researchers using Scopus and Web of Science, focusing on nine marketing and e‐commerce themed research questions. Their findings were consistent with ours, as they concluded that Elicit demonstrated “great potential in surfacing rare literature that other means miss.” Suggesting that the key question is not whether Elicit can replace traditional search methods, but how it can enhance systematic review processes by complementing conventional approaches.

Responsible AI in Evidence Synthesis (RAISE) is a framework that provides tailored recommendations to ‘ensure the responsible, transparent and ethical use of AI’ [111]. There are two principles in the RAISE framework that are particularly important in maintaining research integrity. One of which is transparency. Although Elicit exposes the prompt, screening criteria, recommendations, and relevance scores. The underlying Semantic Scholar query and version history of pre‐edited criteria remains opaque which limits full reproducibility. A fundamental requirement for systematic reviews and similar evidence syntheses ‐ as outlined in methodological guidance such as the Cochrane Handbook [1], the JBI Manual for Evidence Synthesis [112], and the PRISMA reporting guidelines [113] ‐ is that the search strategies must be fully transparent and reproducible. Since Elicit does not provide access to the complete search strategy, the search process cannot be replicated. Consequently, preventing Elicit's use as the primary search method for these types of evidence syntheses. The second salient point is human oversight (also referred to as the ‘human‐in‐the‐loop’). Bhaumik [114], emphasises the human‐centric modality of deployment where the steps are AI‐enabled rather than fully automated. As AI systems lack moral agency, accountability rests with the people who use them. This is essential to ensure trust in both the evidence synthesis and the conclusions made [111], and to guard against the potential responsibility gap if errors do occur. However, without legal authority, RAISE cannot guarantee compliance. This highlights the need for formal government regulation of AI tools to protect both patients and researchers. Bhaumik [114], puts it very well, “ technology, much like a cat, cannot be put back in the bag once it is out. Let us put a bell to ensure that the cat does not run wild.”

There are a number of limitations of our study. The nature of our study was retrospective as we were analysing Elicit after the evidence syntheses had been carried out. Although we did aim to take into account the difference in timing of the searches using Elicit and the traditional searches, this was not always possible. Also, due to Elicit's lack of history of the pre‐edited criteria, we could not quantify the exact change in the number of eligible studies caused by the minor adjustments we made to the inclusion criteria. Additionally, Elicit is being constantly updated as “if you asked once before the update and once after, you might get two different answers from two different version of Elicit.” [115] This will be a limitation of any evaluation of the performance of Elicit, as the results and conclusions may be different as new versions of Elicit are released.

Strengths of our study include the use of four reviews in different areas of health research which enhanced the generalizability and robustness of our findings. We were also able to identify Elicit's strengths and limitations, and spot patterns with how Elicit works for different topics. Furthermore, there is a lack of prior research on Elicit for systematic searches. So, this study is important in quantifying the sensitivity and precision of Elicit which can help in deciding if Elicit currently is a viable research tool. Further research could include: (1) conducting sensitivity analyses using additional studies identified by Elicit, and (2) exploring whether there is any bias in the types of studies Elicit includes compared with those included by human reviewers.

5. Conclusions

This study highlights the high precision of the current version of Elicit AI, making it useful for writing grant proposals, and conducting scoping and preliminary searches. The unique studies identified in Elicit demonstrate that it can be a useful adjunct to traditional searching, but it is not equipped to replace standard systematic searching due to its low sensitivity across the four reviews. Further independent evaluations of Elicit are needed as Elicit is updated and newer versions are released.

Author Contributions

Oscar Lau: investigation, formal analysis, writing – review and editing, writing – original draft. Su Golder: conceptualization, methodology, investigation, writing – original draft, writing – review and editing, supervision.

Ethics Statement

The authors have nothing to report.

Conflicts of Interest

The authors declare no conflicts of interest.

1. Peer Review

The peer review history for this article is available at https://www.webofscience.com/api/gateway/wos/peer-review/10.1002/cesm.70050.

Lau O., and Golder S., “Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies,” Cochrane Evidence Synthesis and Methods 3 (2025): 1‐12, 10.1002/cesm.70050.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.