Artificial Intelligence Search Tools for Evidence Synthesis: Comparative Analysis and Implementation Recommendations

ABSTRACT

To inform implementation recommendations for novel or emerging technologies, Research Information Services at Canada's Drug Agency conducted a multimodal research project involving a literature review, a retrospective comparative analysis, and a focus group on 3 Artificial Intelligence (AI) or automation tools for information retrieval (AI search tools): Lens.org, SpiderCite, and Microsoft Copilot. For the comparative analysis, the customary information retrieval practices used at Canada's Drug Agency served as our reference standard for comparison, and we used the eligible studies of 7 completed projects to measure tool performance. For searches conducted with our usual practice approaches and with each of the 3 tools, we calculated sensitivity/recall, number needed to read (NNR), time to search and screen, unique contributions, and the likely impact of the unique contributions on the projects’ findings. Our investigation confirmed that AI search tools have inconsistent and variable performance for the range of information retrieval tasks performed at Canada's Drug Agency. Implementation recommendations from this study informed a “fit for purpose” approach where Information Specialists leverage AI search tools for specific tasks or project types.

1. Introduction

Novel automation or Artificial Intelligence (AI) applications for information retrieval (AI search tools) have the potential to expedite evidence synthesis tasks, such as study identification or search strategy development [1, 2]. These AI search tools, including generative AI Large Language Model (LLM) chatbots, require performance testing and validation to inform implementation recommendations for Canada's Drug Agency and other evidence synthesis producers.

Canada's Drug Agency is a pan‐Canadian health organization, created and funded by Canada's federal, provincial, and territorial governments to provide independent evidence and advice so health system leaders can make informed drug, health technology, and health system decisions [3]. The Research Information Services (RIS) team at Canada's Drug Agency consists of credentialled Information Specialists (ISs) who conduct systematic and abbreviated literature searches for evidence reports produced by the agency.

There is a long history of automation in the information sciences [4] and we recognize AI as an umbrella term for technology tools that perform tasks that would ordinarily require biological brainpower to accomplish [5]. Our definition of AI search tools is deliberately broad to include successive generations of technologies for information retrieval and to recognize the potential value of older, developing, emerging, or novel tools.

In an earlier project phase, RIS evaluated 51 AI search tools and ranked 17 using our evaluation instrument [6]. In this successive phase, RIS tested the performance of 3 top‐ranked tools to support the work of Canada's Drug Agency:

• Lens.org (“The Lens”) https://www.lens.org

• SpiderCite https://sr-accelerator.com/#/spidercite

• Microsoft Copilot https://www.microsoft.com/en-ca/microsoft-copilot

Our objective in testing these tools was to determine if they should replace or supplement current RIS information retrieval methods given the estimated contribution of eligible and unique studies, and the resources needed to conduct searches and screen the results. As these tools use different automation technologies to perform different search tasks, our investigation also aimed to compare their distinct strengths and weaknesses.

1.1. Lens.org

Lens.org is a freely available metadata aggregator of scholarly and patent literature. Lens.org has ingested, aggregated, and normalized records for over 272 million scholarly works and over 155 million patent records [7]. RIS identified Lens.org's Scholarly Search as a promising information source to capture unique, relevant references not identified by our usual practice search approaches.

1.2. SpiderCite

SpiderCite generates all forwards (cited by) and backwards (citing) citations (1 level deep) based on an input library of articles which were chosen for inclusion in a study [8]. SpiderCite sources its data from Lens.org and has been incorporated into the TERA suite of automation tools for systematic reviews (https://tera-tools.com/spidercite).

We selected SpiderCite as a reliable tool for citation searches, a supplementary technique recommended for topics that are difficult to search [9].

1.3. Microsoft Copilot

Microsoft Copilot is a generative AI chatbot that uses LLMs and was formerly known as Bing Chat [10]. Copilot utilizes a model built upon OpenAI's GPT‐4 LLM5 and has an interface similar to the ChatGPT chatbot studied for search strategy development [2]. Microsoft Copilot offered RIS an opportunity to test a LLM with additional data security through the agency's Office 365 license and Microsoft Copilot's enterprise data protection [11].

2. Materials and Methods

Our primary investigation method was a retrospective comparative analysis. We also conducted a literature review to better understand the uses of SpiderCite, Lens.org, and Copilot in current information retrieval practice. Further, we conducted a post‐test focus group to enrich our results data by collecting accounts from the ISs about their experiences conducting test searches with these tools. The full protocol for this study is available in our Supporting Information: [Link], [Link], [Link].

2.1. Literature Review

An information specialist (MW) searched PubMed Central, Google Scholar, and Europe PubMed Central (Europe PMC) for published evidence reviews that reported using Lens.org, SpiderCite, and/or Copilot for their search methods. We chose these sources as they search the full text of records, since we expected AI search tools to be described in the methods sections of reports and not in the titles, abstracts, or indexing. The complete search strategies for PubMed Central are presented in Appendix 1; other search strategies are available on request. The main search concepts were the 3 tools and evidence synthesis or literature searching. The records from the Google Scholar search results were captured using the Publish or Perish program (https://harzing.com/resources/publish-or-perish/). All searches were run on February 3, 2025. To capture only material published since the inception of the tools, searches for Copilot were limited to results published between January 1, 2023, to the present, and searches for Lens.org were limited to results published between January 1, 2000, to the present. Searches for SpiderCite were not limited by date as the search terms were unlikely to be used in reference to other concepts than the tool itself. Results were compiled in EndNote 21 for assessment and duplicate removal.

We excluded results that reported using the tools only for non‐information retrieval tasks. We also excluded protocols, non‐English language publications, and articles published before the inception dates of the tools (2023 for Copilot, 2000 for Lens.org, and 2021 for SpiderCite). Finally, we excluded preprints when the published versions were also retrieved, in order not to bias the results by capturing the same data twice.

We analyzed the selected reviews and generated a narrative summary of how Lens.org, SpiderCite, and Copilot have been used for information retrieval by evidence synthesis producers.

2.2. Comparative Analysis

For our comparative analysis of tool performance, we compared reference searches (i.e., using usual practice sources and approaches) and test searches (i.e., using each of the AI search tools) for 7 completed agency projects. An IS conducted the reference and test searches for each project, and each project was assigned a different IS.

2.2.1. Sample Projects

We identified a sample of 7 projects published in the preceding 24 months with a range of health topics and review methodologies to represent the scope of agency reports (Table 1). Three of the projects utilized rapid or abbreviated search methods (including an update and a qualitative review); 1 project was an overview of reviews (i.e., a review of systematic reviews, or “umbrella review”); 2 used systematic search approaches; and 1 was a pragmatic search for authors on a specified condition (author search). Author searches are a common request for RIS, and we wanted to test the performance of the tools for this distinct information retrieval task (i.e., not a systematic or abbreviated search for an evidence synthesis report).

Table 1.

Selected projects.

ID	Search method	Type	Topic	Title	Selected articles
1: RC1504	Abbreviated/Rapid	Simple	Device	Radiofrequency ablation for chronic knee, hip, and shoulder pain	9
2: RC1522	Abbreviated/Rapid	Simple	Drug	Ketamine for adults with treatment resistant depression or PTSD: a 2023 update	14
3: OP0553	Overview of reviews	Complex	Health system	ED overcrowding—Overview of reviews	64
4: RC1476	Abbreviated/Rapid	Simple	Procedure	Interventions for the presurgical management of knee osteoarthritis: a rapid qualitative review	16
5: HC0043	Systematic	Complex	Health system	Considerations of access and inclusion in adolescent eating disorder care	21
6: HC0080	Systematic	Complex	Health system	Culturally appropriate care for the treatment or management of substance or opioid use	15
7: SR0827	Author search	Simple	Condition	Clinical author identification: Schizophrenia	15

The finalized reports for selected evidence synthesis projects needed to include at least 5 selected articles incorporated into the reviews' findings. Author search results typically include 10–20 Canadian authors who have recently published on a clinical intervention, so we used a recent project that had requested a search to identify 15 authors.

We selected 3 projects with topics that we classified as “Simple” (i.e., with defined interventions and indications), and 3 projects with “Complex” topics (i.e., with undefined or many possible interventions or indications). For the presentation of our findings, we separated the results for the author search (even though we also classified it as a “Simple” topic type).

2.2.2. Reference Searches

The search approaches that were implemented when the projects were originally undertaken served as our reference standard for comparison. To retrieve comparable results given the time lag between when the project searches were originally conducted, ISs repeated these “reference searches” in the original databases on the same date as test searches. Given the resource requirement to repeat grey literature searches, we reused grey literature results from when the reference searches were originally conducted and accounted for this discrepancy in our analysis of unique contributions from each of the test searches.

2.2.3. Conducting Test Searches in Lens.org

For Projects 1 through 6, ISs executed an abbreviated/simplified search in Lens.org based on concept sets and search filters from the strategies used for the reference searches.

For the simple author search (Project 7), an IS executed an abbreviated/simplified strategy for the project's indication with search filters and analytics to identify Canadian authors.

2.2.4. Conducting Test Searches in SpiderCite

For Projects 1 through 6, we imported into SpiderCite selected studies from their published reports. Using SpiderCite, we generated references “cited by” (i.e., the references that were cited by the selected studies, also known as backwards citations) and “citing” (i.e., the references that cite the selected studies, also known as forward citations) those studies. Generated references constituted our test results for comparison. We reported 2 sets of results: SpiderCite—Cited By, and SpiderCite—Citing.

We excluded Project 7 from our tests of SpiderCite. As simple author searches do not have selected studies, citation searches could not be conducted for this project type.

2.2.5. Conducting Test Searches Using Copilot

For Projects 1–6, the ISs generated PubMed search strategies using similar prompt queries as described by Wang et al. [2]: For a systematic review titled, [title], can you generate a systematic review Boolean search strategy to find all included studies in PubMed on the review topic? ISs evaluated the strategy generated by Copilot and provided further, iterative prompts to focus or expand concept sets in the generated Boolean strategies. Once the ISs felt satisfied that the Copilot strategies met our expectations for comprehensiveness and precision, they executed the Copilot generated strategies in PubMed and exported the results to compare against MEDLINE results from the reference searches.

For the simple author search (Project 7), an IS generated a PubMed search strategy using a modified single prompt query: Can you generate a Boolean search strategy to find studies in PubMed about [indication] by Canadian authors published in the last five years? The IS evaluated the Copilot strategy and provided further prompts when needed. As described above, we executed the Copilot generated strategy in PubMed and exported the results to compare against the reference search.

2.2.6. Data Collection

The ISs recorded data from reference and test searches in Excel. These data are available in our supplementary files. To facilitate data analysis and comparison, ISs exported search results to EndNote libraries on an internal shared drive. One IS (DM) then compared search results from reference and test searches in EndNote to determine how many eligible studies were captured by each approach (i.e., sensitivity).

For reference searches, we calculated:

• 1.Sensitivity or recall (the estimated performance of a search, calculated as the number of eligible articles retrieved divided by the number of eligible articles).
• 2.Number needed to read (NNR) (the estimated number of articles needed to read to find one eligible article, calculated as 1 divided by the positive predictive values, or the total number of articles retrieved by the search divided by the number of eligible articles).
• 3.Time in hours to search (including time to execute, export, and deduplicate; excluding search development).
• 4.Time in hours to screen the results based on estimate of 3.33 h to screen 100 items [12] [(total # of articles retrieved divided by 100) × 3.33].

  For test searches, we calculated the same values above and:

• 5.Unique contributions (recorded as the number of unique results, not captured by the reference searches).
• 6.Likelihood that unique contributions could impact the findings.

As the Copilot test searches retrieved only PubMed/MEDLINE results, we adjusted the sensitivity and NNR calculations using the number of eligible MEDLINE articles from the reference searches.

A statistician (EM) calculated Clopper‐Pearson confidence intervals (CIs) for sensitivity using the binom.test () function in the software R version 4.2.1. A 95% CI was constructed for sensitivity to provide a range of plausible values.

NNR was calculated using the delta method for a log‐transformed Wald‐type CI, which is more stable when probabilities are small and ensures that estimates are positive [13]. All NNR values were rounded up to the nearest whole number.

2.2.7. Determining Unique Contributions

An IS (DM) used EndNote to isolate unique studies captured by the test searches (i.e., unique contributions) and screened titles and abstracts against the selection criteria in the project reports. The IS evaluated only the first 50 items (most recently published first, then sorted alphabetically by author) if there were more than 50 unique contributions from a test search. Based on the IS's evaluation, we estimated the likelihood that unique contributions would impact the findings.

We classified the unique contributions in our data collection spreadsheet as:

• Very unlikely: 0 items meet criteria for further assessment.

• Unlikely: < 2 items meet criteria for further assessment.

• Likely: 2–3 items meet criteria for further assessment.

• Very likely: ≥ 4 items meet criteria for further assessment.

To account for using the original grey literature results as part of our reference searches (i.e., and not rerunning grey literature searches), we evaluated unique contributions published after our original search dates from the test searches. We excluded from our analysis any unique contributions published after our original search date and likely to have been found by our standard approaches for grey literature searching (i.e., from included sources in our internal grey literature checklists).

2.3. Focus Group

We held a 60‐min, semi‐structured focus group on February 11, 2025, with 7 RIS ISs. A RIS Manager (RF) facilitated a discussion with the questions in Appendix 2 and recorded the session. ISs received the questions a day before the focus group and, after the facilitated discussion, had an extra week to add written responses. We reviewed the recording and the written responses and narratively summarized the themes that emerged.

3. Results

3.1. Literature Review

After duplicates were removed, the searches retrieved a combined 981 results. After screening, a total of 378 results were retained that used one of the 3 tools for information retrieval purposes. Of these, 329 were retrieved by the Lens.org searches, 41 by the SpiderCite searches, and 8 by the Copilot searches.

Of the 329 total reviews that used Lens.org in their search methods, nearly half (149; 45%) were patent reviews (also known as patent landscape reviews or patent landscape analyses). The large number of patent reviews is unsurprising as Lens.org was initially created as a patent database [14]. Of the rest of the Lens.org reviews, 109 (33%) were standard evidence syntheses (systematic reviews, scoping reviews, meta‐analyses, etc.). A total of 35 (11%) were other types of reviews, mainly bibliographic analysis, and 4 reviews (1%) used Lens.org for citation searching, but did not specify whether this was through another tool like CitationChaser or through the website itself. Finally, 32 reviews (10%) used Lens.org to conduct brief, non‐systematic searches, such as finding the number of patents on a given topic where that information was used only for a subsection of the paper and not as the main purpose of the study. The remainder of the Lens.org analysis focuses only on the 109 evidence syntheses.

3.1.1. Usage With Other Sources

As part of understanding the usage of these tools, we analyzed whether each review used the tool as the sole source for information retrieval, or whether it was used in combination with other tools or sources.

In the case of Lens.org, of the 109 evidence syntheses, Lens.org was the only database used in 25 reviews (23%). A total of 79 reviews (72%) searched Lens.org in combination with other databases or search engines, which ranged widely from medical databases like MEDLINE to broad interdisciplinary databases like Scopus or Google Scholar. The remaining five reviews (5%) were unclear about their sources.

As a corpus of existing articles is required to use SpiderCite, it is unsurprising that no reviews used SpiderCite as the only information retrieval tool—although one review was unclear about exactly which other sources were used [15].

Likewise, there were no reviews in which Copilot was the only tool used, and in fact it was always used as a secondary tool in addition to traditional database searching.

3.1.2. Field of Study

Although evidence synthesis is predominantly produced within the field of health sciences, its use is growing in other fields [16]. In our analysis, all but 1 of the 41 SpiderCite reviews were on health topics (98%); the remaining review was on a computer science topic, but even this review overlapped with healthcare to the extent that it searched PubMed [17]. This finding may reflect that SpiderCite was developed by a team working within evidence‐based healthcare [18] and has been promoted in the health science sphere, such as through mentions in guidance like the TARCiS statement and JBI guidance [9, 19].

By contrast, Lens.org and Copilot were more mixed, with half of the Copilot reviews and 42% of the Lens.org reviews being on topics in the health sciences. Neither of these tools was developed for use in health sciences, and they are not as well established within evidence‐based medicine as SpiderCite.

3.1.3. Tool‐Specific Observations

Lens.org is the most similar of the three tools to existing sources for information retrieval in evidence syntheses. As a database that uses familiar query syntax, its use was similar to other databases such as MEDLINE or Scopus. In contrast, 9 reviews (8%) used Lens.org as an adjunct to standard database searching. This was primarily for grey literature searching, in the case of 5 (5%) of the reviews [20, 21, 22, 23, 24]. Other individual uses were to find articles within a specific journal [25], to locate updates to previously retrieved articles [26], to conduct a search update [27], and in one case to be used in an unspecified “supplementary” way [28].

Unlike Lens.org and Copilot, there is little variation in how SpiderCite can be used. The 2 decisions made when using SpiderCite are (1) which functions (citing/forward citation and cited by/backward citation) to use, and (2) which articles to input as seed articles (e.g., included studies for the review or key articles found during scoping). Of the 41 SpiderCite reviews, we could not determine how SpiderCited had been used for 7 (17%) as in the case where a review reported that “SpiderCite […] w[as] also used where appropriate” [29]. Of the remaining 34 reviews, the majority (29; 70%) used both the forward citation and backward citation searching functions. Four articles (10%) used only backward citation searching and 1 (2%) used only forward citation searching. Fourteen of the reviews (34%) did not specify which seed articles the SpiderCite search was conducted on. Of the remainder, 18 (44%) used SpiderCite on included studies for the review, and 9 (22%) on other sets of articles, such as all full‐text review articles, seed articles, and various other studies deemed relevant.

In general, of the three tools, Copilot demonstrated both the most variety and the least transparency in how it was used. It was also used in very few reviews comparatively, which makes it more difficult to draw general conclusions. Of the 8 Copilot reviews, 3 (37.5%) were vague about how Copilot was used, simply saying that it was used for searching [30, 31, 32] (e.g.,: “Copilot was used as a search assistant” [31]). In 2 others (25%), we understand that Copilot was used to identify specific studies, although in both cases the reporting was unclear [33, 34].

The remaining 3 reviews (37.5%) described their use of Copilot more explicitly [35, 36, 37]. Muona and Podina et al. both used Copilot to generate synonyms. Muona used Copilot throughout the research process, not only for the search component; in terms of searching, they explicitly asked Copilot for Scopus keywords, and also for tips and guidance on building a search in Scopus [35]. Both Copilot prompts and responses are presented in a published appendix to help the reader understand how the tool was used. The author also reported that, “[b]efore conducting the final SCOPUS search, an initial search criteria were tested with the assistance of Copilot,” but it is unclear what this refers to [35].

In Podina et al. [36], the authors used ChatGPT 4o, Perplexity AI, and Copilot to develop keywords, including MeSH terms, and to create strategies. In their appendix, they include what appear to be the responses of the different tools but not the authors' prompts, although they do describe their general question in the text [36]. Copilot's response includes multiple separate search strings which contain MeSH as well as keyword terms, implying that these searches were designed for PubMed. The authors do not provide their final search strategy, so it is unclear which of the different tools' responses informed the search.

Finally, Robinson et al. [37] was the only review to use Copilot for search translation. The authors used Copilot to translate the MEDLINE search strategy into Embase, Cochrane, and Google Scholar. In addition, Copilot was also used to “refine the search,” after which the strategy was reviewed by the authors [37]. Although the final search strategies are provided, there is no further detail on how Copilot was used.

While none of the reviews used Copilot as the sole source for information retrieval, Copilot was the only genAI tool used specifically for searching in 6 reviews (75%). Of those 6, 3 (37.5%) did not use other genAI tools at all [31, 35, 37], while the remaining 3 reviews (37.5%) used other genAI tools but for non‐search purposes (ChatGPT and Claude for Nogueira [32], deepL for Stucky [33], and ChatGPT for Weisman [34]). Two reviews (25%) used additional genAI tools for searching; Lumbiganon used ChatGPT and Bard [30], while Podina used ChatGPT and Perplexity, as mentioned above [36].

3.2. Comparative Analysis

To test the performance of these tools, ISs executed reference and test searches for the 7 sample projects between August 21, 2024, and November 29, 2024. The team conducted 33 searches in total and analyzed the results to generate the findings summarized below. We grouped results by complexity for Projects 1 through 6 and presented results separately for the simple author search (Project 7).

3.2.1. Sensitivity

The average sensitivity of reference searches for Projects 1–6 was 0.986 (95% CI: 0.952–0.998). For simple projects and the simple author search, reference searches had a sensitivity of 1 (95% CI: 0.907–1). For complex projects, reference searches performed with a sensitivity of 0.98 (95% CI: 0.93–0.998).

Lens.org had the highest sensitivity of our test searches with an average of 0.676 (95% CI: 0.594–0.75) for Projects 1–6. Lens.org performed better for simple projects and the simple author search with an average sensitivity of 0.816 (95% CI: 0.657–0.923) and score of 0.9 (95% CI: 0.555–0.997), respectively. For complex projects, Lens.org's sensitivity was 0.6 (95% CI: 0.497–0.697).

SpiderCite had comparatively low sensitivity for Projects 1–6. The tool performed with an average value of 0.23 (95% CI: 0.165–0.306) for Cited By searches and an average value of 0.257 (95% CI: 0.189–0.335) for Citing searches. Sensitivity performance was comparable for simple and complex searches: 0.263 (95% CI: 0.134–0.431) and 0.24 (95% CI: 0.16–0.336) for Cited By Searches, and 0.289 (95% CI: 0.154–0.459) and 0.27 (95% CI: 0.186–0.368) for Citing searches. We could not test SpiderCite for the simple author search.

Copilot had the widest performance range: from 0 (95% CI: 0–0.206) for Project 4 to 0.909 (95% CI: 0.587–0.998) for Project 2. The tool performed with an average value of 0.244 (95% CI: 0.172–0.328) for Projects 1–6. Copilot performed better for simple projects and the simple author search with sensitivity values of 0.412 (95% CI: 0.246–0.593) and 0.5 (95% CI: 0.187–0.813), respectively. For complex projects, Copilot's sensitivity was 0.145 (95% CI: 0.077–0.239). See Table 2 and Figure 1 for complete sensitivity data.

Table 2.

Sensitivity by project for reference and test searches.

Projects	Sensitivitya
	Reference	Lens.org	SpiderCite—Cited by	SpiderCite—Citing	Copilot
Simple projects
1: RC1504	1 (95% CI: 0.631–1)	0.75 (95% CI: 0.349–0.968)	0.125 (95% CI: 0.003–0.527)	0.25 (95% CI: 0.032–0.651)	0.571 (95% CI: 0.184–0.901)
2: RC1522	1 (95% CI: 0.768–1)	0.857 (95% CI: 0.572–0.982)	0.143 (95% CI: 0.018–0.428)	0.143 (95% CI: 0.018–0.428)	0.909 (95% CI: 0.587–0.998)
4: RC1476	1 (95% CI: 0.794–1)	0.812 (95% CI: 0.544–0.96)	0.438 (95% CI: 0.198–0.701)	0.438 (95% CI: 0.198–0.701)	0 (95% CI: 0–0.206)
Average	1 (95% CI: 0.907–1)	0.816 (95% CI: 0.657–0.923)	0.263 (95% CI: 0.134–0.431)	0.289 (95% CI: 0.154–0.459)	0.412 (95% CI: 0.246–0.593)
Complex projects
3: OP0553	1 (95% CI: 0.944–1)	0.688 (95% CI: 0.559–0.798)	0.234 (95% CI: 0.138–0.357)	0.297 (95% CI: 0.189–0.424)	0.079 (95% CI: 0.026–0.176)
5: HC0043	0.952 (95% CI: 0.762–0.999)	0.429 (95% CI: 0.218–0.66)	0.238 (95% CI: 0.082–0.472)	0.238 (95% CI: 0.082–0.472)	0.286 (95% CI: 0.037–0.71)
6: HC0080	0.933 (95% CI: 0.681–0.998)	0.467 (95% CI: 0.213–0.734)	0.267 (95% CI: 0.078–0.551)	0.2 (95% CI: 0.043–0.481)	0.385 (95% CI: 0.139–0.684)
Average	0.98 (95% CI: 0.93–0.998)	0.6 (95% CI: 0.497–0.697)	0.24 (95% CI: 0.16–0.336)	0.27 (95% CI: 0.186–0.368)	0.145 (95% CI: 0.077–0.239)
Average (Projects 1–6)	0.986 (95% CI: 0.952–0.998)	0.676 (95% CI: 0.594–0.75)	0.23 (95% CI: 0.165–0.306)	0.257 (95% CI: 0.189–0.335)	0.244 (95% CI: 0.172–0.328)
Simple author search
7: SR0827	1 (95% CI: 0.692–1)	0.9 (95% CI: 0.555–0.997)	NA	NA	0.5 (95% CI: 0.187–0.813)

^a Sensitivity (recall) is the ratio of eligible articles to the total number of eligible articles retrieved.

^b Average values are presented in bold for evidence synthesis projects (1–6), showing differences between simple and complex projects.

Figure 1.

Sensitivity estimates with 95% confidence intervals. Sensitivity values for simple projects are displayed along the top row, and complex projects are displayed along the bottom row. Values for the simple author search are shown in Figure 2.

3.2.2. Number Needed to Read

The average NNR of reference searches for Projects 1–6 was 83 (95% CI: 71–98). For simple projects, reference searches had an average NNR of 61 (95% CI: 45–84); and for the simple author search, the NNR was 33 (95% CI: 18–61). For complex projects, reference searches performed with a NNR of 97 (95% CI: 80–118).

Compared to our reference searches, Lens.org had a slightly higher NNR on average for Projects 1–6 of 98 (95% CI: 81–119). Again, Lens.org performed better for simple projects and the simple author search with an average NNR of 100 (95% CI: 70–141) and a score of 12 (95% CI: 6–21), respectively. For complex projects, Lens.org's NNR was 110 (95 CI: 86–142).

The NNR values for SpiderCite varied for Cited by and Citing searches with averages for Projects 1–6 of 54 (95% CI: 39–75) and 131 (95% CI: 96–180), respectively. For simple projects, the NNR for Cited by searches was much lower at 27 (95% CI: 15–49) compared to 158 (95% CI: 88–285) for Citing searches. With complex projects, the average NNR values were closer at 65 (95% CI: 44–97) for Cited by searches and 120 (95% CI: 83–175) for citing searches.

Copilot also had the widest performance range for NNRs: from an incalculable score for Project 4 (because none of the eligible articles were captured) to a score of 537 (95% CI: 135–2145) for Project 5. The tool performed with an average NNR of 83 (95% CI: 59–118) for Projects 1–6. Again, Copilot performed better for simple projects and the simple author search with values of 36 (95% CI: 22–60) and 16 (95% CI: 7–36), respectively. For complex projects, Copilot's NNR was 167 (95% CI: 95–292). See Table 3 and Figure 3 for complete NNR values.

Table 3.

Number needed to read (NNR) estimates by project for reference and test searches.

Projects	Number needed to read (NNR)a
	Reference	Lens.org	SpiderCite—Cited by	SpiderCite – Citing	Copilot
Simple projects
1: RC1504	76 (95% CI: 39–152)	56 (95% CI: 26–123)	77 (95% CI: 11–540)	662 (95% CI: 166–2645)	35 (95% CI: 14–92)
2: RC1522	63 (95% CI: 38–106)	182 (95% CI: 104–319)	12 (95% CI: 4–44)	83 (95% CI: 21–330)	36 (95% CI: 20–66)
4: RC1576	52 (95% CI: 32–84)	44 (95% CI: 26–76)	24 (95% CI: 12–49)	36 (95% CI: 18–74)	NEb
Average	61 (95% CI: 45–84)	100 (95% CI: 70–141)	27 (95% CI: 15–49)	158 (95% CI: 88–285)	36 (95% CI: 22–60)
Complex projects
3: OP0553	70 (95% CI: 55–89)	31 (95% CI: 23–41)	70 (95% CI: 43–116)	124 (95% CI: 79–193)	19 (95% CI: 8–44)
5: HC0043	147 (95% CI: 95–228)	242 (95% CI: 126–464)	61 (95% CI: 26–144)	145 (95% CI: 61–348)	537 (95% CI: 135–2145)
6: HC0080	149 (95% CI: 88–250)	439 (95% CI: 210–919)	53 (95% CI: 20–138)	55 (95% CI: 18–168)	166 (95% CI: 69–396)
Average	97 (95% CI: 80–118)	110 (95% CI: 86–142)	65 (95% CI: 44–97)	120 (95% CI: 83–175)	167 (95% CI: 95–292)
Average (Projects 1–6)	83 (95% CI: 71–98)	98 (95% CI: 81–119)	54 (95% CI: 39–75)	131 (95% CI: 96–180)	83 (95% CI: 59–118)
Simple author search
7: SR0827	33 (95% CI: 18–61)	12 (95% CI: 6–21)	NA	NA	16 (95% CI: 7–36)

^a Number needed to read (NNR) is the estimated number of articles needed to read to find one eligible article. It is calculated as the total number of articles retrieved by the search divided by the number of eligible articles. NNR is rounded up to the nearest whole number.

^b NE = not estimable.

^c Average values are presented in bold for evidence synthesis projects (1–6), showing differences between simple and complex projects.

Figure 3.

Number needed to read (NNR) estimates for reference and test searches with 95% confidence intervals. NNR values for simple projects are displayed along the top row, and complex projects are displayed along the bottom row. Values for the simple author search are shown in Figure 2.

Figure 2.

Sensitivity and NNR with 95% confidence intervals for Project 7—SR0827. Sensitivity values for the simple author search are displayed in the left‐hand graph, and NNR values are displayed in the right‐hand graph. We did not calculate values for SpiderCite searches for this project type.

3.2.3. Correlation: Sensitivity and NNR for Reference and Test Searches

To illustrate how the tools performed for all projects (including Project 7—the simple author search) we correlated the data to generate Figure 4. Figure 4 shows both estimates (sensitivity and NNR) and their uncertainty (CIs). While the NNR for the SpiderCite—Cited by and Copilot searches was better or comparable to our reference searches, these tools performed with much lower sensitivity. Lens.org sensitivity was closer to our reference searches, but the tool had a higher NNR. SpiderCite—Citing searches performed with lower sensitivity and a higher NNR.

Figure 4.

Correlation of sensitivity and NNR for reference and test searches. Correlated sensitivity and NNR values for all projects are shown with their confidence intervals.

3.2.4. Time to Search

ISs recorded the time to search in hours to execute searches, export, and deduplicate results. We excluded search development tasks (e.g., the time to develop the search strategy based on consultations and review of any topic initiation documents) as our retrospective study design meant this study happened before our investigation.

For our reference, the average time to search for Projects 1–6 was 2.88 h. This represents a wide range of times, from 30 min for Project 6 to 8.5 h for Project 5. The average time to search for simple projects was lower at 1.75 h compared to 4 h for complex projects.

Lens.org searches took longer to execute than any of our other test tools with an average time to search for Projects 1–6 of 2.25 h. We did not observe the same pattern of simple searches taking less time to search. Simple searches took an average of 2.75 h to execute in Lens.org and complex searches took only 1.75 h.

SpiderCite had predictable times to search for Cited By and Citing searches, reflecting that the same seed articles generated both sets of results. The average time to search for Projects 1–6 was 1.25 and 1.29 h, respectively. We also did not observe a difference between simple and complex projects, except for Project 3 which used 64 seed articles to execute the test searches. This project had more than 3 times as many seed articles as any of the other projects, so we anticipated this extra time was needed to build the record set to import into SpiderCite and to deduplicate the results.

Copilot had the lowest time to search with an average of 0.96 h across all projects. We observed that the average for simple searches was much faster at 0.67 h compared to the average for complex projects at 1.25 h.

The reference time to search for Project 7 (the simple author search) was lower at 1 h than either of the comparative searches for Lens.org or Copilot at 2 and 1.75 h, respectively. See Table 4 for complete data.

Table 4.

Recorded time to search by project for reference and test searches.

Projects	Time to search in hoursa
	Reference	Lens.org	SpiderCite—Cited by	SpiderCite—Citing	Copilot
Simple projects
1: RC1504	1	1	1.5	1.75	0.75
2: RC1522	3.5	2.5	1.25	1.25	0.75
4: RC1576	0.75	4.75	0.5	0.5	0.5
Average	1.75	2.75	1.08	1.17	0.67
Complex projects
3: OP0553	3	2	2.75	2.75	1
5: HC0043	8.5	2	1	1	1.75
6: HC0080	0.5	1.25	0.5	0.5	1
Average	4	1.75	1.42	1.42	1.25
Average (Projects 1–6)	2.88	2.25	1.25	1.29	0.96
Simple author search
7: SR0827	1	2	NA	NA	1.75

^a Time to search is calculated in hours as the time to execute searches, export, and deduplicate results. Search development tasks were excluded from the time estimate as our retrospective study design meant this study happened before our investigation.

^b Average values are presented in bold for evidence synthesis projects (1–6), showing differences between simple and complex projects.

3.2.5. Time to Screen

We estimated the time to screen using an established benchmark value for systematic reviews of 3.33 h to screen the title and abstracts of 100 items [12].

For our reference searches, the average time to screen for Projects 1–6 was 65.3 h. We calculated that simple projects take less time to screen on average at 25.63 h compared to complex projects at 104.97 h.

The time to screen estimates for Lens.org were the highest for any of our test searches with an average of 53.64 h for Projects 1–6. We estimated that search results from Lens.org would take longer to screen for complex projects at 73.12 h compared to simple projects at 34.17 h. Interestingly, the time to screen estimate for simple searches in Lens.org was higher than our reference.

The time to screen estimates for SpiderCite search results were all lower than our reference values. We observed that the time to screen estimates were lower for Cited By searches at 10.08 h compared to Citing searches at 27.57 h. We also observed the same pattern as with our reference searches and the other tools: time to screen estimates were all less for simple projects compared to complex ones.

Copilot also had a low time to screen estimate with an average of 13.84 h across all projects. However, there was a wide range from 0.23 h for Project 4 to 35.76 h for Project 5. Simple projects also had a shorter time to screen estimate at 5.55 h compared with complex projects at 22.12 h.

We did not calculate time to screen values for Project 7 (the simple author search) as screening for this project type is conducted by the IS according to different criteria as evidence synthesis projects. However, as the reference NNR for Project 7 was higher at 33 than the NNR for Lens.org at 12 and Copilot at 16, it follows that the time to screen values would be lower for the test searches. See Table 5 for the complete time to screen estimates.

Table 5.

Time to screen estimates by project for reference and test searches.

Projects	Time to screen in hoursa
	Reference	Lens.org	SpiderCite—Cited by	SpiderCite—Citing	CoPilot
Simple projects
1: RC1504	20.25	11.09	2.56	44.09	4.63
2: RC1522	29.17	72.43	0.77	5.53	11.79
4: RC1576	27.47	18.98	5.43	8.23	0.23
Average	25.63	34.17	2.92	19.28	5.55
Complex projects
3: OP0553	148.09	44.86	34.77	77.99	3.10
5: HC0043	97.70	72.36	10.02	24.14	35.76
6: HC0080	69.13	102.13	6.96	5.46	27.51
Average	104.97	73.12	17.25	35.86	22.12
Average (Projects 1–6)	65.30	53.64	10.08	27.57	13.84
Simple author search
7: SR0827	NA	NA	NA	NA	NA

^a Time to screen is calculated in hours as the time to evaluate the titles and abstracts of the search results generated by each tool. It is based on estimate of 3.33 h to screen 100 items.

^b Average values are presented in bold for evidence synthesis projects (1–6), showing differences between simple and complex projects.

3.2.6. Unique Contributions

By comparing the results of our test searches against the reference searches in EndNNote, an IS (DM) isolated the unique articles captured only by the test searches (i.e., the unique contributions). Table 6 shows the numbers of unique contributions captured by each test search.

Table 6.

Unique contributions.

Projects	Unique articles not captured by the reference searchesa
	Lens.org	SpiderCite—Cited by	SpiderCite—Citing	Copilot
Simple projects
1	160	50	1284	62
2	1484	10	103	12
4	238	127	203	1
Average	627	62	530	25
Complex projects
3	648	978	2205	68
5	1575	248	597	859
6	2435	162	127	514
Average	1553	463	976	480
Average (Projects 1–6)	1090	263	753	253
Simple author search
7	13	NA	NA	11

^a The number of unique article records compared against the reference standard (i.e., the number of results not captured by the reference searches).

^b Average values are presented in bold for evidence synthesis projects (1–6), showing differences between simple and complex projects.

Every test search captured unique contributions, but the searches conducted in Lens.org captured the most with an average of 1090 unique contributions for Projects 1–6. SpiderCite Citing searches also retrieved many unique contributions with 753 on average for Projects 1–6. SpiderCite—Cited By and Copilot searches captured far less with 263 and 253 items, respectively.

3.2.7. Likelihood Unique Contributions Could Impact Findings

Given the time and cost to screen unique contributions, we estimated the likelihood that these articles could impact the findings of the projects (i.e., would they be worth the effort to evaluate). We classified the unique contributions for each test search as shown in Table 7.

Table 7.

Likelihood unique contributions could impact findings.

Projects	Likelihood that unique articles not captured by the reference searches could impact the project's findingsa
	Lens.org	SpiderCite—Cited by	SpiderCite—Citing	Copilot
Simple projects
1	Likely: 2–3 items meet criteria for further assessment	Likely: 2–3 items meet criteria for further assessment	Unlikely: < 2 items meet criteria for further assessment	Very likely: ≥ 4 items meet criteria for further assessment
2	Unlikely: < 2 items meet criteria for further assessment	Very unlikely: 0 items meet criteria for further assessment	Very unlikely: 0 items meet criteria for further assessment	Very unlikely: 0 items meet criteria for further assessment
4	Unlikely: < 2 items meet criteria for further assessment	Unlikely: < 2 items meet criteria for further assessment	Very unlikely: 0 items meet criteria for further assessment	Very unlikely: 0 items meet criteria for further assessment
Complex projects
3	Likely: 2–3 items meet criteria for further assessment	Very unlikely: 0 items meet criteria for further assessment	Very likely: ≥ 4 items meet criteria for further assessment	Very likely: ≥ 4 items meet criteria for further assessment
5	Very likely: ≥ 4 items meet criteria for further assessment	Very likely: ≥ 4 items meet criteria for further assessment	Very likely: ≥ 4 items meet criteria for further assessment	Very likely: ≥ 4 items meet criteria for further assessment
6	Very likely: ≥ 4 items meet criteria for further assessment	Very likely: ≥ 4 items meet criteria for further assessment	Very likely: ≥ 4 items meet criteria for further assessment	Very unlikely: 0 items meet criteria for further assessment
Simple author search
7	Likely: 2–3 items meet criteria for further assessment	NA	NA	Very likely: ≥ 4 items meet criteria for further assessment

^a The likelihood that unique contributions could impact the findings was estimated by screening the first 50 items (most recently published) according to the projects' selection criteria. We classified the contributions as “Very unlikely” (0 items meet criteria for further assessment); “Unlikely” (fewer than 2 items meet criteria for further assessment); “Likely” (2–3 items meet criteria for further assessment); and “Very likely” (4 or more items meet criteria for further assessment).

^b Values for “Very likely” are presented in bold, showing differences between simple and complex project types.

For simple projects, we classified the unique contributions from only 1 test search as being very likely to impact the project's findings. Conversely, we classified the unique contributions from 9 test searches as being very likely to impact the findings for complex projects, including all 3 test searches for complex projects conducted using SpiderCite—Citing.

3.3. Focus Group

Themes from our focus group of ISs are presented below by tool, followed by general impressions of testing AI search tools for the comparative analysis.

3.4. Lens.org

ISs described translating complex queries in Lens.org as tedious and expressed a general lack of trust that complex searches in Lens.org had performed as intended. They reported frustration at Lens.org's 2000‐word character limit and at the requirement to add field codes before every keyword phrase in the Query Text Editor. ISs also noted limitations of the Structured Search interface, including a lack of language limits or an option to focus subject headings. ISs reported that Lens.org contained low quality records that made deduplication tasks more time‐consuming. Finally, ISs reported that PDFs included in the export file took a long time to process, particularly for large results sets. Multiple ISs compared searching Lens.org to searching Google Scholar and recommended simplifying queries and avoiding many synonyms.

Without being informed of the results of the comparative analysis, ISs felt that Lens.org was potentially useful for specific device topics and for identifying Canadian authors. The checkbox to limit results by geographic area was mentioned as a particularly valuable feature. One IS reported using this feature after the comparative analysis to look for potential clinical experts for another project.

3.4.1. SpiderCite

ISs described the “Cited By” and “Citing” searches in SpiderCite as quick and simple to execute. SpiderCite results generated as expected for a citation searching tool. While ISs recognized the utility of citation searching, they expressed uncertainty about integrating this type of searching into agency workflows and could only foresee using SpiderCite when seed articles were available at the beginning of the project. Otherwise, ISs would need to identify seed articles based on their best judgment or emerging guidance about how many articles to include and how to avoid bias.

ISs reported being unable to use SpiderCite for the simple author search. SpiderCite also did not work well for an overviews of reviews project (i.e., where search results are limited to systematic reviews). Results contained trials and other study designs that needed to be removed manually. Date limits also posed a problem for “Cited By” searches and ISs reported capturing results outside of the date range limits for projects.

Further, because SpiderCite requires DOIs, the tool does not generate results for most grey literature seed articles. This reliance on DOIs reduces the potential applications of the tool for agency projects that primarily reference grey literature. As SpiderCite's data comes from Lens.org, ISs recounted similar problems retrieving poor quality records that took longer to deduplicate.

When asked about the potential uses for SpiderCite, ISs shared that the tool would be useful for complex review topics when seed articles are known at the beginning of the project.

3.4.2. Copilot

Reports varied from ISs about using Copilot to generate a PubMed search strategy. While one IS described Copilot as, “kind of fun,” and another said that it, “worked fine,” other ISs called it, “my nemesis,” and described prompting the tool as, “the most frustrating process I've ever been through.”

In some instances, the initial Copilot strategies were too focused and required prompts to expand each concept. Conversely, some initial Copilot strategies were too broad and needed additional prompts to focus each concept. Copilot generated some useful synonym terms but also suggested unhelpful terms for concepts like “eating disorders” and “overcrowding.”

ISs also described writing precisely detailed prompts to generate the desired results. When they prompted the chatbot to add field codes or MeSH terms, “[…] it didn't add them in the right way,” and “[i]t was just like, here's your original strategy and some MeSH terms without combining them.” When prompting Copilot to generate a PubMed query for a complex topic, one IS reported that they used so many prompts that they caused Copilot to time‐out.

The number of search results for Copilot‐generated strategies was also widely variable. One IS recounted “fixing” the strategy because it retrieved over 400,000 records in PubMed. Another IS reported the opposite problem of having far too few results (less than 200 when they were expecting over 1000).

ISs also described a time‐consuming process to validate the strategies that Copilot generated. Even after providing exact instructions and validating the results, most ISs edited manually the search strategy generated by Copilot because it, “wasn't good enough to run in PubMed.” ISs reported removing quoted phrases generated by Copilot that retrieved zero results in PubMed and excluding bad terms (e.g., “therapy”) that caused unreasonably high numbers of results.

Further, ISs expressed concern about Copilot‐generated strategies without expert‐operator validation. Because Copilot's confidently conversational style obscured suboptimal results, a non‐expert could easily assume that a Copilot search strategy will produce adequately sensitive search results for a project. The ISs' impressions from the focus group (confirmed by the comparative analysis' quantitative results) were that Copilot‐generated strategies performed with low sensitivity—even when the strategies were developed with the prompts and validation of expert operators.

ISs reported that Copilot performed well to generate a PubMed search query for a single concept. Ironically, ISs felt that these types of simple tasks (e.g., literal translation) do not require assistance. When asked about functions of Copilot that they did like, ISs recounted instances where Copilot suggested some good synonym terms (e.g., for “underserved populations,” and “cultural competence”) and that Copilot would be particularly helpful for initial search development and general scoping tasks.

ISs also described unexpected or surprising uses of Copilot. Through testing, ISs discovered that Copilot could append an agency filter to a search strategy without first being provided with the filter. Copilot also helped us improve a geographic filter by providing a large list of domains for Canadian institutions and agencies.

3.4.3. General Impressions

Overall, ISs expressed surprise that the usefulness of the AI search tools depended on the project type (e.g., simple vs complex, device vs health system questions, etc.). Tools would work very well for a specific project type but then terribly (or not at all) for another project. For example, Lens.org worked very well for a simple author search while SpiderCite could not generate results at all for this type of search. Copilot could translate a simple intervention concept into PubMed syntax but did not work well for complex projects, while SpiderCite “Citing” searches worked very well for these complex projects.

ISs expressed enthusiasm for experimenting with novel AI search tools but a preference for trusted tools. One IS shared that they would be “open to testing the tools again on a good day when I have time.” And while a couple of ISs reported applying AI search tools for project work unrelated to this investigation, consensus was that ISs are more likely “to return to the tools that they know and trust and understand.”

4. Discussion

4.1. Strengths and Limitations

It was a strength of the comparative analysis that we measured tool performance using the eligible articles from real agency projects. It may have introduced bias that the ISs who ran the test searches knew the “target” articles. However, any directed prompting of the tools to yield the desired outcomes would have only increased the sensitivity of the test searches. Knowing this, readers are cautioned that the 3 AI search tools may not perform as well in a real world setting when the target articles are not known to the searchers in advance. A further strength of this study was that professional ISs designed the methods and tested each of the tools against high‐quality reference standard search methods also executed by professional ISs.

While we selected projects with a range of health topics and search approaches, it is a limitation of the comparative analysis that our sample includes only 7 agency projects and tested 3 tools. As these projects reflect the range of health topics undertaken by a national health technology assessment agency, our findings may not be generalizable to the full scope of potential evidence synthesis topics. Also, while our observations about these tools may be useful for both expert and non‐expert searchers, our recommendations are intended for similar audiences of ISs working in evidence synthesis production. Further, we did not compare the performance for AI search tools used in combination (e.g., sensitivity for Lens.org plus Copilot). Combining AI search tools may be worth exploring, although we anticipate for these 3 tools that Lens.org's sensitivity would be comparable to its recall when used in combination with other tools. Due to the metadata aggregator's size and its data being used for SpiderCite, it is unlikely that SpiderCite or Copilot could capture additional eligible articles not included in the Lens.org results.

4.2. Observations About the Tools

The AI tools we tested are all quite different from one another and are discussed separately, followed by some general and comparative findings.

4.2.1. Lens.org

While Lens.org indexes a vast volume of literature, much larger than MEDLINE or Embase, our focus group findings were that the search platform does not easily allow for complex searches, making it a poor fit for systematic searching. Nonetheless, our literature review found that Lens.org is being used for this purpose. Of the three AI tools tested, Lens.org had the highest sensitivity, with much better sensitivity for simple searches. The NNR for Lens.org was higher compared to our reference searches and again performed better with simple searches. Time to search was long for Lens.org, owing to the difficulties in adapting to the inadequacies of the search platform. At Canada's Drug Agency, the database could be used for simple searches on very specific topics, including device reviews or author searches, when our standard search practices are not able to retrieve an adequate volume of information.

4.2.2. SpiderCite

Since SpiderCite performs straightforward citation (Cited By and Citing) searching on a set of seed articles, it is both easy to use and can be a helpful supplemental search for complex topics. Guidance on how to use SpiderCite does exist, and while not all reviews in our literature review clearly reported how the tool was used, it is possible to use it in reproducible and clearly described ways. The primary question with SpiderCite is whether it is required to supplement standard searching. In our testing, SpiderCite had comparatively low sensitivity for all projects. The NNR was variable, low for Cited By but quite high for Citing. Unique contributions were ranked as being very likely to impact the findings for 5 of the 12 SpiderCite searches, including all 3 test searches for complex projects conducted using SpiderCite—Citing, indicating its potential usefulness for these types of projects. Based on these results, SpiderCite could only be considered as a supplement to our standard searching due to low sensitivity. For our purposes a citation tracking tool, such as SpiderCite, could supplement usual practice information retrieval for complex search topics where seed articles are known to the information specialist at the beginning of a project. It could potentially be used mid‐project, after the initial list of included studies has been compiled, but incorporating a second set of search results at this point in the review would create significant process issues.

4.2.3. Copilot

As a generative AI chatbot, Copilot could potentially be used at multiple steps in the literature search process. Based on our literature review, Copilot's uses for information retrieval in evidence synthesis are the least consistent and transparent. While 8 reviews used Copilot in their search methods, none were explicit enough in their description that a reader could replicate that use. However, as we understand that LLMs, such as Copilot, are not useful for generating lists of included studies for systematic reviews [38, 39], we tested Copilot on its ability to create search strategies that could then be performed by ISs.

Findings from our comparative analysis and our focus group showed that an individual's knowledge of searching, familiarity with the topic under review, and prompt engineering skills were required to achieve best results from Copilot. Without the knowledge of expert searchers, Copilot offers search strategies of unpredictable quality that that miss relevant studies or yield large number of irrelevant results. Skillful prompt engineering and IS knowledge allows searches to be refined in a more usable way.

But even with prompts from expert searchers, Copilot varied widely in performance, with comparatively low sensitivity and NNR values ranging from very low to very high. Due to low sensitivity and variable NNR, it cannot be recommended for the creation of ready‐to‐run search strategies. However, Copilot or another LLM can be adopted to identify keyword terms for search strategy development. For some projects, it was adept in assisting ISs in this task. Future skills development by ISs in prompt engineering may also help improve Copilot's performance for information retrieval tasks.

4.3. General Observations

Our investigation demonstrated to us how AI search tools can give users a false sense of confidence. Skillful prompt engineering is required to produce optimal results when developing search strategies for information retrieval using generative AI, and experience in evidence reviews is needed to validate AI‐generated strategies and their search results. It is only when we know from experience what to expect from database search results that we can adjust a malfunctioning strategy. Further, an understanding of search syntax and construction is needed to search databases with limited functionality such as Lens.org. As with citation searching tools, like SpiderCite, users must select seed articles with care to avoid biasing the results. Users must also determine appropriate review topics for citation search tools and understand the limitations of the results.

Such an understanding of the individual strengths and weaknesses of these tools is critical when using the tools for information retrieval in evidence synthesis, and we risk generating imprecise results when we assume the tools can perform without an experienced operator. Additionally, the risk increases as more information retrieval tasks are performed by the tool. For example, a tool like SpiderCite performs a specific task and poses less risk because the operator is responsible for selecting the seed articles and deciding which results to generate. While a generative AI tool like Copilot could confidently provide the researcher with seemingly accurate and complete search results with very little operator input.

Our findings also showed us that the 3 AI tools sometimes retrieved unique results that we considered very likely to impact project findings, especially for complex topics. And while the use of these AI tools for searching and screening sometimes achieved a direct time saving compared to the standard reference search, they did so at a great cost to sensitivity and/or NNR. For some complex topics, potential increases in the likelihood of retrieving additional studies may be worth the increased time to search and screen incurred by supplemental use of these AI tools by an expert operator. We also acknowledge that the performance of a generative AI tool, like Copilot, is dependent on prompt engineering skills. Future educational investment by ISs in building effective prompts for LLMs is likely to improve the performance of these tools for information retrieval.

4.4. Implementation Recommendations

Based on this investigation, RIS proposed the implementation recommendations below for Canada's Drug Agency. As these recommendations are specific to our agency, they may not be appropriate for all evidence synthesis producers. However, our general observations about the variable and operator‐dependent performance of AI search tools will likely support a similar “fit for purpose” approach for ISs looking to implement automation or AI technologies into their information retrieval tasks.

RIS recommended supplementing current information retrieval methods with AI search tools when it is likely these tools will improve sensitivity (recall) or retrieve unique eligible research records without increasing unreasonably the NNR and/or the cost to screen results.

For the specific tools that we tested, we recommend:

• 1.Searching Lens.org for simple device reviews.
• 2.Searching Lens.org to identify potential authors to serve as clinical experts (i.e., for simple author searches) when the minimum number of target experts cannot be retrieved from usual practice approaches.
• 3.Supplementing usual practice information retrieval with citation searching using SpiderCite or an equivalent tool for complex search topics when seed articles are known to the information specialist at the beginning of a project.
• 4.Consulting Copilot or another LLM to identify keyword terms for search strategy development.

Implementation of these AI search tools for evidence syntheses should be informed by an understanding that the tools may behave inconsistently and are not appropriate for all project types.

Beyond the specific implementation recommendations for the 3 tested tools, RIS investigations reinforced fundamental conditions for any AI search tools for evidence synthesis products. AI search tools should only be used with:

• 1.An informed understanding about how the tool works and when it is likely to benefit a project's findings without placing an unreasonable burden on the project team.
• 2.A skilled operator with the ability to correct the tool's operation and to validate its performance.

Beyond RIS' plans to incorporate AI search tools into our information retrieval approaches according to the recommendations above, we will also continue to monitor and test promising AI search tools in the future. Given the fast pace of technology development, we will reinstate our evaluation approaches as novel AI or automation tools become available to us.

5. Conclusion

This study shows that the 3 AI search tools under investigation (Lens.org, SpiderCite, and Copilot) are not currently suitable for creating database strategies for complex systematic searches approaches due to wide variations in sensitivity or unreasonable increases in NNR. However, these and similar AI search tools do show promise in assisting ISs in specific aspects of this process, including keyword identification, very simple search construction, and supplemental citation searching, but only under the guidance of an expert searcher. As the field of AI is rapidly expanding, Canada's Drug Agency will continue to monitor tool development and evaluate novel AI search tools for information retrieval tasks.

Author Contributions

Robin Featherstone: conceptualization, methodology, writing – original draft, writing – review and editing, project administration, supervision, investigation, formal analysis, data curation, validation, resources. Melissa Walter: writing – original draft, writing – review and editing, investigation, methodology, conceptualization, data curation, formal analysis, visualization. Danielle MacDougall: methodology, software, data curation, investigation, validation, formal analysis, project administration, writing – review and editing, conceptualization. Eric Morenz: software, visualization, formal analysis, writing – review and editing. Sharon Bailey: investigation, writing – review and editing, data curation. Robyn Butcher: investigation, writing – review and editing, data curation. Caitlyn Ford: investigation, writing – review and editing, data curation. Hannah Loshak: investigation, writing – review and editing, data curation. David Kaunelis: writing – review and editing, writing – original draft, conceptualization, methodology, data curation, investigation.

Conflicts of Interest

The authors declare no conflicts of interest.

Peer Review

The peer review history for this article is available at https://www.webofscience.com/api/gateway/wos/peer-review/10.1002/cesm.70045.

Supporting information

Supporting Information S1: Supplement Data Comparative‐Analysis‐of‐AI‐Search‐Tools.

Supporting Information S2: Supplement Protocol Comparative‐Analysis‐of‐AI‐Search‐Tools.

Supporting Information S3: RFeatherstone CESM+Declaration+of+Interest FORM.

Appendix 1. Search Strategies for PubMed Central

Lens.org

(lens.org[All Fields] OR www.lens.org[All Fields] OR “https://www.lens.org”[All Fields] OR “http://www.lens.org”[All Fields] OR “lens database”[All Fields] OR “lens databases”[All Fields] OR “lens website”[All Fields] OR “lens search engine”[All Fields] OR “patent lens”[All Fields] OR (“the lens”[All Fields] AND “scholarly search”[All Fields]) OR “the lens platform”[All Fields] OR “lens patent database”[All Fields])

AND

(((systematic[ti] OR scoping[ti] OR umbrella[ti]) AND (review[ti] OR overview[ti] OR synthesis[ti])) OR Technology assessment[ti] OR (patent[ti] AND (landscape[ti] OR review[ti] OR analysis[ti])) OR health technology assessment[ti] OR HTA[ti] OR meta analysis[ti] OR metaanalysis[ti] OR integrative review[ti] OR integrative literature review[ti] OR quantitative synthesis[ti] OR methodological review[ti] OR methodological overview[ti] OR hand search[ti] OR systematic search[ti] OR Comprehensive Literature Review[ti] OR patent review[ti] OR rapid review[ti] OR evidence review[ti] OR evidence synthesis[ti] OR systematic review[ab] OR systematic literature review[ab] OR scoping review[ab] OR scoping literature review[ab] OR umbrella review[ab] OR health technology assessment[ab] OR HTA[ab] OR meta analysis[ab] OR metaanalysis[ab] OR systematic overview[ab] OR integrative review[ab] OR integrative literature review[ab] OR quantitative synthesis[ab] OR methodological review[ab] OR methodological overview[ab] OR hand search[ab] OR systematic search[ab] OR Comprehensive Literature Review[ab] OR patent review[ab] OR rapid review[ab] OR evidence review[ab] OR evidence synthesis[ab] OR systematic review[article] OR systematic literature review[article] OR scoping review[article] OR scoping literature review[article] OR umbrella review[article] OR health technology assessment[article] OR HTA[article] OR meta analysis[article] OR metaanalysis[article] OR systematic overview[article] OR integrative review[article] OR integrative literature review[article] OR quantitative synthesis[article] OR methodological review[article] OR methodological overview[article] OR hand search[article] OR systematic search[article] OR Search strateg*[article] OR search string*[article] OR database search*[article] OR search term*[article] OR searching term*[article] OR patent search*[article] OR literature search*[article] OR Comprehensive Literature Review[article] OR patent review[article] OR rapid review[article] OR evidence review[article] OR evidence synthesis[article] OR (Review[ti] AND (search[article] OR searching[article] OR searched[article])) OR Systematic Reviews as Topic[mh] OR Meta‐Analysis as Topic[mh] OR Network Meta‐Analysis as Topic[mh] OR Technology Assessment, Biomedical[mh])

SpiderCite

Spidercite*[All Fields] OR “spider cite”[All Fields]

Copilot

((microsoft[All Fields] AND copilot[All Fields]) OR (bing[All Fields] AND copilot[All Fields]) OR (microsoft[All Fields] AND “co‐pilot”[All Fields]) OR (bing[All Fields] AND “co‐pilot”[All Fields]) OR BingChat[All Fields] OR “Bing Chat”[All Fields] OR “Bing AI Chat”[All Fields] OR ((“AI”[ti] OR “artificial intelligence”[ti] OR GenAI[ti] OR conversation agent*[ti] OR conversational agent*[ti] OR chatbot*[ti]) AND copilot[All Fields]))

AND

(((systematic[ti] OR scoping[ti] OR umbrella[ti]) AND (review[ti] OR overview[ti] OR synthesis[ti])) OR Technology assessment[ti] OR (patent[ti] AND (landscape[ti] OR review[ti] OR analysis[ti])) OR health technology assessment[ti] OR HTA[ti] OR meta analysis[ti] OR metaanalysis[ti] OR systematic overview[ti] OR integrative review[ti] OR integrative literature review[ti] OR quantitative synthesis[ti] OR methodological review[ti] OR methodological overview[ti] OR hand search[ti] OR systematic search[ti] OR Search strateg*[ti] OR search string*[ti] OR database search*[ti] OR search term*[ti] OR searching term*[ti] OR patent search*[ti] OR literature search*[ti] OR Comprehensive Literature Review[ti] OR patent review[ti] OR rapid review[ti] OR evidence review[ti] OR evidence synthesis[ti] OR hand search[article] OR systematic search[article] OR Search strateg*[article] OR search string*[article] OR database search*[article] OR search term*[article] OR searching term*[article] OR patent search*[article] OR literature search*[article] OR Systematic Reviews as Topic[mh] OR Meta‐Analysis as Topic[mh] OR Network Meta‐Analysis as Topic[mh] OR Technology Assessment, Biomedical[mh]).

Appendix 2. Questions for post‐test focus group

What were your impressions of the Lens.org search?

What did you like about Lens.org?

Under what circumstances would you use Lens.org again?

What were your impressions of SpiderCite?

How would you compare the “Cited by” and the “Citing” analyses? Are you more likely to use one or the other in the future?

Under what circumstances would you use SpiderCite again?

What were your impressions of Copilot?

What did you like about Copilot?

Did anything surprise you about the searches that you conducted for the project?

What advice would you give to another IS interested in using any of these tools?

Data Availability Statement

The data that supports the findings of this study are available in the Supporting Information of this article.