Evaluating the Performance of Covidence's Artificial Intelligence‐Based Screening Function in Cancer Systematic Reviews

ABSTRACT

Background

Systematic reviews (SRs), often conducted for evidence synthesis, require extensive multistage screening, with Stage‐I title/abstract screening being especially time consuming. Covidence systematic review software, a widely used SR platform, has offered artificial intelligence (AI)‐assisted Stage‐I screening function since 2022. However, this function's performance has not yet been formally compared across heterogeneous, large‐scale oncology‐focused SRs.

Methods

We selected two completed SRs conducted by the Program in Evidence‐based Care (PEBC), Ontario, Canada. The first SR (Axilla‐BC, N = 8774) supported a clinical practice guideline in breast cancer and the second SR (PET‐utility, N = 7267) informed the provincial Positron Emission Tomography (PET) steering committee in Ontario. Using each SR, we conducted a simulation‐based study by drawing subsets of 500, 1000, and 2000 articles and running 30, 30, and 10 simulation trials, respectively, to emulate Covidence's AI‐assisted Stage‐I screening. For each trial, we calculated the workload and time savings at 95% and 100% sensitivity for relevant articles and at 100% sensitivity for finally included articles as primary outcomes. Secondary outcomes comprised missed finally included articles at 95% sensitivity for relevant articles. Wilcoxon rank‐sum tests were used to compare outcomes between these two SRs.

Results

When pooling all subset sizes and the N = 1000 subsets, workload savings at 95% sensitivity were significantly higher for Axilla‐BC than for PET‐utility (median = 37.4% vs. 26.2%, W = 3156.5, p = 0.003 and median = 38.3% vs. 20.5%, W = 641.0, p = 0.005, respectively). Corresponding time savings were also significantly higher for Axilla‐BC (median = 6.0 vs. 4.2 h, W = 3089.5, p = 0.008 and median = 9.6 vs. 5.1 h, W = 639.0, p = 0.005). In Axilla‐BC, 20% of the 70 trials missed a total of 18 finally included articles, but only one PET‐utility trial missed one article.

Conclusions

In two large oncology‐SRs, Covidence's AI‐assisted Stage‐I screening function demonstrated meaningful potential to reduce reviewer workload and screening time. Nonetheless, Covidence's performance varied notably between reviews, suggesting a possible association between the results and the characteristics of the SR, including topic, methodological complexity, relevance rate, and inclusion rate. Careful human oversight remains essential to ensure that no potentially finally included studies are missed in evidence synthesis, particularly in nuanced or methodologically complex reviews.

In two large oncology‐SRs, Covidence systematic review software's Artificial Intelligence (AI)‐assisted title and abstract screening function demonstrated meaningful potential to reduce reviewer workload and screening time. Nonetheless, Covidence's performance varied notably between reviews, suggesting that results may depend on the SR topic and complexity. Careful human oversight remains essential to ensure that no potentially finally included studies are missed, particularly in nuanced or methodologically complex reviews.

Abbreviations

AI

artificial intelligence

CPG

clinical practice guidelines

IQR

interquartile range

PEBC

Program in Evidence‐Based Care

PET

positron emission tomography

SR

systematic review

1. Introduction

Evidence synthesis is the systematic process of identifying, appraising, and integrating findings from a range of sources to inform policy, practice, and decision‐making for policymakers and interest‐holders [1]. High‐quality evidence synthesis requires a well‐formulated research question and rigorous, methodical execution throughout all the stages of the systematic review (SR) process. A central component of this process is a comprehensive literature search for retrieving everything relevant to addressing the research questions. However, comprehensive search could result in the retrieval of a high proportion of irrelevant articles that need to be systematically excluded, through the process of screening, before the subsequent stages of synthesis. A manual screening process can be very time‐consuming. For example, the mean time taken to complete an SR has been estimated at 67.3 weeks [2], driven by manual screening. The development of SR‐based clinical practice guidelines (CPGs) can take approximately 2 years, largely due to the time‐intensive requirements of conducting SRs and the need to adhere to rigorous methodological and reporting standards [3, 4]. To improve efficiency in evidence synthesis, the design and application of artificial intelligence (AI)‐based tools are becoming more common [5, 6]. However, a recent study [7] also found that the usage of these tools in evidence synthesis is reported inconsistently and is mostly confined to screening tasks among the four review stages examined in the study (search, screening, data extraction, and analysis/synthesis).

Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia), one of the most popular platforms [8] that supports article screening, introduced its AI‐assisted title and abstract screening (Stage‐I screening) function in 2022 [9]. However, formal evaluations of Covidence's AI performance in cancer‐related evidence synthesis are rare [10, 11], despite its large user‐base from more than 450 licensed institutions. The gap is notable given the continuous and rapid growth of oncological medical literature and the increasing demand for timely, high‐quality evidence to inform clinical and policy decisions.

We previously published a study [10] on a simulation‐based evaluation assessing the performance of Covidence's AI‐assisted Stage‐I screening function in the context of a complex breast cancer CPG developed by the Program in Evidence‐Based Care (PEBC) under Ontario Health (Cancer Care Ontario), Ontario, Canada [12]. Our study demonstrated that the function offered some workload and time savings, while also revealing several limitations, including variability across trials and the potential risk of missing finally included articles when sensitivity thresholds were relaxed. These findings highlighted both the promise and the pitfalls of applying AI‐assisted screening to a methodologically complex, narrowly focused oncology SR.

To assess the generalizability of these findings, we subsequently applied the same simulation‐based methods to a second SR from PEBC. This SR supported a 6‐month evidence synthesis report examining the clinical utility of Positron Emission Tomography (PET) mainly across multiple cancer types. The objective of our study was to compare the workload and screening time savings generated by Covidence's AI‐assisted Stage‐I screening function by applying a simulation‐based evaluation to these two previously completed cancer‐related SRs. By directly comparing results across these two SRs, this study also aimed to provide insights into how a review's scope and complexity may influence the performance of AI‐assisted screening in oncology SRs and evidence‐based CPGs.

2. Materials and Methods

2.1. Data sets

This study is based on two previous SRs developed by PEBC [12]. The first SR informed a published CPG focused on axillary management in early‐stage breast cancer [13] and was used in a prior study [10] evaluating the AI‐assisted screening function in Covidence (henceforth, referred to as Axilla‐BC). The second SR focused on the clinical use of PET across multiple cancer types as well as in selected non‐oncological conditions based on Ontario Health's (Cancer Care Ontario) requirement, including sarcoidosis, epilepsy, or dementia, and informed the provincial PET steering committee in Ontario, Canada (henceforth, referred to as PET‐utility) [14].

For the Axilla‐BC study, 8774 articles were considered for Stage‐I Screening, the relevance rate (defined as percentage of articles found relevant after Stage‐I screening) is 8.8%, and inclusion rate (defined as percentage of relevant articles that were finally included for synthesis) is 14%. For the PET‐utility study, 7267 articles were considered for Stage‐I Screening, the relevance rate is 2.9%, and the inclusion rate is 30.8%. Important characteristics that indicate the variations and complexities of these two data sets are provided in Table 1.

Table 1.

Characteristics of systematic reviews considered in our study.

SR name	Number of research questions	Number of disease entities	Eligibility criteria*	Relevance rate (total number of articles before Stage‐I screening)	Inclusion rate (total number of articles after Stage‐I screening)
Axilla‐BC	5	1 (BC)	Women or men with early‐stage BC, studies with > 100 patients undergoing axillary staging (e.g., SLNB, ALND, ultrasound staging) and treatment strategies (e.g., surgical and radiotherapy interventions), primary studies published between 2007 and February 2020.	8.8% (8774)	14% (774)
PET‐utility	1	4 (cancer, noncardiac sarcoidosis, epilepsy, and dementia)**	Adult and pediatric populations related to the four diseases, prospective studies with ≥ 12 patients or retrospective studies with ≥ 50 patients (≥ 25 for sarcoma) evaluating PET/CT or PET/MRI, studies published during January ‐ June 2024	2.9% (7267)	30.8% (208)

Abbreviations: ALND, axillary lymph node dissection; BC, breast cancer; CT, computed tomography; MRI, magnetic resonance imaging; PET, positron emission tomography; SLNB, sentinel lymph node biopsy; SR, systematic review.

^*Only major criteria are listed; see the individual SR for detailed inclusion/exclusion criteria.

^**Only one finally included study was related to non‐cancerous disease, i.e., dementia.

From each SR, three subsets of articles (N = 500, 1000, and 2000) were randomly sampled without replacement from the Stage‐I screening articles of Axilla‐BC and PET‐utility and were uploaded as title‐abstract pairs to Covidence for the simulation process (described in the next section). Figure 1 shows the number of relevant and finally included articles in these subsets for both SRs. All subsets preserved the relevance (8.8% and 2.9%, respectively) and inclusion rates (14% and 30.8%, respectively) for Axilla‐BC and PET‐utility.

Figure 1.

Number of relevant and finally included articles in the full Axilla‐BC (N = 8774) and PET‐utility (N = 7267) systematic reviews, and in their randomly generated subsets (N = 500, 1000, 2000). BC, breast cancer; PET, positron emission tomography.

2.2. Simulation Procedure and Trials

Each simulation began with a predefined subset of articles (e.g., N = 500 for Axilla‐BC). The same protocol was followed for both Axilla‐BC and PET‐utility to enable comparison. In each simulation, the first 25 articles (including at least 2 relevant articles and 2 irrelevant articles) were manually screened by a reviewer to allow the AI algorithm to begin learning (i.e., machine‐learning training), as per Covidence's requirement. After this step, a typical 10–20‐min pause (as recommended by the Covidence support team and per the alignment with our experience from numerous pilot trials conducted before developing the study protocol) was taken to allow time for Covidence's machine‐learning algorithm to update and for a system notification to appear, indicating that machine learning had been applied to revise the relevance of the remaining articles. Based on the revised relevance, Covidence automatically sorted the articles from highest to lowest predicted relevance, with the more relevant articles displayed at the top of the list. Human reviewers worked through this AI‐ranked list sequentially, accepting or rejecting each article based on whether the article had been included or excluded in the original Stage‐I screening (i.e., human reviewers were not blinded to the original inclusion/exclusion decision). As screening progressed, Covidence's active learning algorithm periodically updated the order of the remaining articles based on the human reviewer's decisions. This process continued until all relevant articles (100%) within the subset were identified, marking the completion of a simulation. The 100% identification of relevant articles guarantees 100% identification of finally included articles.

Each completed simulation was treated as a separate trial. To accommodate natural variability in machine learning behavior resulting from the variation in training data (i.e., the set of initial 25 articles), multiple trials, each with a different set of initial 25 articles from the corresponding subset, were conducted. For the subsets with N = 500 and N = 1000 articles, 30 trials were conducted. For the subset with N = 2000 articles, the number of trials was restricted to 10 due to the substantial time taken by human reviewers (approximately, 1.6, 4.5, and 10 h for N = 500, 1000, and 2000, respectively). In addition, any screening error (e.g., errors resulting from the accidental wrong selection of the relevant/irrelevant indicator as informed by reviewers of the original SR) required a trial to be restarted from the beginning. Therefore, in this retrospective, simulation‐based study, it was necessary for the current reviewers to be unblind to the original screening decisions.

For each of the 70 trials, a new project was created in Covidence using the corresponding subset. The order of the articles was randomly assigned for each project by Covidence. However, the first 25 articles in each trial might not always include 2 relevant ones required for machine‐learning training. While screening the initial 25 articles, the human reviewers followed Covidence's randomly assigned order with one exception. If after screening 23 (or 24) irrelevant papers, there were 0 (or 1), that is, less than 2 relevant articles among them, the human reviewers needed to randomly find and add 2 (or 1) relevant articles to satisfy Covidence's machine‐learning training requirement. This procedure resulted in a near‐random selection of the initial 25 articles but provided no assurance that trials were comparable regarding screening difficulty or the distribution of topics represented by the articles.

The human reviewers also ensured that the system notification appeared after screening the initial set. In general, the notification appeared within 20 min after completion of the initial 25‐article screening.

2.3. Outcome Measures

The primary outcomes were workload savings and time savings. Workload savings is defined as the proportion of articles that did not require screening by a human reviewer (unscreened) to identify a certain proportion of studies (either relevant or finally included) in a trial at a predefined threshold of sensitivity for the identification of relevant articles or finally included articles. Time savings is the number of hours that would have been taken to screen unscreened articles by a human reviewer only, assuming a human reviewer screens 40 studies per hour [15], at a predefined threshold of sensitivity. Actual time savings can vary based on the skill/experience of the human reviewer and the subject matter of review. The rate of 40 studies per hour is also supported by PEBC's internal reports [12] and our published protocol [3] on a related topic. We calculated workload savings and time savings at 95% sensitivity for relevant articles (92.9% in N = 500 PET‐utility subset, considering one missed identification in the smallest subset, which had only 14 relevant articles), and 100% sensitivity for relevant articles, as well as 100% sensitivity for finally included articles.

The secondary outcomes comprised the proportion of the finally included articles that were missed by the AI‐assisted screening function at the 95% (or 92.9% in N = 500 PET‐utility subset) sensitivity for relevant articles and the proportion of relevant articles screened when all finally included articles were identified.

2.4. Statistical Analysis

We compared the workload savings and time savings separately in the subset‐specific trials from Axilla‐BC and PET‐utility and across all trials using the Wilcoxon rank‐sum test in RStudio [16]. For this test, W denotes the Wilcoxon rank‐sum statistic as reported by R. The proportions of trials with missed finally included articles were compared between the two SRs. All statistical tests were two‐sided, and a significance level of α = .05 was used.

2.5. Ethics Statement

Our study is based on two SRs, one of which supported a published CPG and the other supported a report for a provincial committee. Hence, institutional ethics approval was not applicable.

3. Results

3.1. Workload and Time Savings Between Axilla‐BC and PET‐Utility at Different Sensitivity Thresholds for Relevant Articles

The workload savings (expressed as percentage of subset size) at 95% sensitivity for relevant articles for Axilla‐BC and PET‐utility are shown in Figure 2a. The median workload savings is higher for Axilla‐BC than PET‐utility for N = 500 (40.7% vs. 31.3%, W = 572.0, p = 0.072) and N = 1000 (38.3% vs. 20.5%, W = 641.0, p = 0.005), but similar for N = 2000 (16.6% vs. 17.6%, W = 50.5, p = 1.000). The median workload savings, across all subsets, is significantly higher (37.4% vs. 26.2%, W = 3156.5, p = 0.003) for Axilla‐BC than PET‐utility.

Figure 2.

Workload savings (as percentage of subset size) for Axilla‐BC and PET‐utility datasets at (a) 95% (92.9% for PET‐utility) and (b) 100% sensitivity for relevant articles for the three subsets (N = 500, 1000, and 2000) and across all subsets combined are shown in the boxplots. Within each boxplot, the central bold line within the box represent the median; the lower and upper edges of the box indicate the first (Q1) and third (Q3) quartiles, respectively; the difference between the upper and lower edges of the box represents the interquartile range (IQR); the lower and upper ends of the vertical line represent (Q1 – 1.5 × IQR) and (Q3 + 1.5 × IQR) as lower and upper limits. Any point outside these limits is considered an outlier. An outlier is represented as a black dot. BC, breast cancer; PET, positron emission tomography.

The workload savings at 100% sensitivity for relevant articles are shown in Figure 2b. The median workload savings is higher in the Axilla‐BC compared to PET‐utility for N = 500 (25.0% vs. 15.6%, W = 539.0, p = 0.191) and N = 1000 (17.3% vs. 8.4%, W = 530.0, p = 0.240), but slightly lower for N = 2000 (4.4% vs. 6.7%, W = 33.0, p = 0.218). The median workload savings in Axilla‐BC is higher than that in PET‐utility across all subsets (18.2% vs. 10.4%, W = 2759.0, p = 0.199). However, none of these differences reach statistical significance.

The time savings (in hours) at 95% sensitivity for relevant articles are shown in Figure 3a. The median time savings are higher in Axilla‐BC than PET‐utility for N = 500 (5.1 vs. 4 h, W = 572.0, p = 0.072) and N = 1000 (9.6 vs. 5.1 h, W = 639.0, p = 0.005), but similar for N = 2000 (8.3 vs. 8.8 h, W = 50.5, p = 1.000). The median time savings, across all subsets, is significantly higher (6.0 vs. 4.2 h, W = 3089.5, p = 0.008) for Axilla‐BC than PET‐utility.

Figure 3.

Time savings (hours) for Axilla‐BC and PET‐utility datasets at (a) 95% (92.9% for PET‐utility) and (b) 100% sensitivity for relevant articles for the three subsets (N = 500, 1000, and 2000) and across all subsets combined are shown in the boxplots. Within each boxplot, the central bold line within the box represent the median; the lower and upper edges of the box indicate the first (Q1) and third (Q3) quartiles, respectively; the difference between the upper and lower edges of the box represents the interquartile range (IQR); the lower and upper ends of the vertical line represent (Q1 – 1.5 × IQR) and (Q3 + 1.5 × IQR) as lower and upper limits. Any point outside these limits is considered an outlier. An outlier is represented as a black dot. BC, breast cancer; PET, positron emission tomography.

The time savings at 100% sensitivity for relevant articles are shown in Figure 3b. The median time savings is higher in Axilla‐BC compared to PET‐utility for N = 500 (3.1 vs. 2 h, W = 536.5, p = 0.203) and N = 1000 (4.4 vs. 2.1 h, W = 532.0, p = 0.228), but lower for N = 2000 (2.2 vs. 3.3 h, W = 33.0, p = 0.218). The median time savings in Axilla‐BC is not significantly different from that in PET‐utility in any of the subsets or across all subsets (3.3 vs. 2.0 h, W = 2711.5, p = 0.277). Again, none of these differences reach statistical significance.

3.2. Workload and Time Savings Between Axilla‐BC and PET‐Utility at 100% Sensitivity for Finally Included Articles

The workload savings and time savings at 100% sensitivity for finally included articles are shown in Figures 4a,b, respectively. The median workload savings is higher in Axilla‐BC compared to PET‐utility for N = 500 (57.6% vs. 53.5%, W = 485.5, p = 0.605) and N = 1000 (63.9% vs. 55.1%, W = 543.5, p = 0.169), but significantly lower for N = 2000 (17.9% vs. 41.8%, W = 21.0, p = 0.029). The corresponding median time savings are higher in Axilla‐BC compared to PET‐utility for N = 500 (7.2 vs. 6.7 h, W = 484.5, p = 0.615) and N = 1000 (16 vs. 13.8 h, W = 542.0, p = 0.176), but significantly lower for N = 2000 (8.9 vs. 20.9 h, W = 21.0, p = 0.029). When results were pooled across all subsets, neither workload savings (60.3% vs. 52.8%, respectively, W = 2561.0, p = 0.645) nor time savings (8.9 vs. 9.8 h, W = 2302.0, p = 0.539) differed significantly between Axilla‐BC and PET‐utility at 100% sensitivity for finally included articles.

Figure 4.

(a) Workload savings (as percentage of subset size) and (b) time savings (hours) for Axilla‐BC and PET‐utility data sets at 100% sensitivity for finally included articles for the three subsets (N = 500, 1000, and 2000) and across all subsets combined are shown in the boxplots. Within each boxplot, the central bold line within the box represent the median; the lower and upper edges of the box indicate the first (Q1) and third (Q3) quartiles, respectively; the difference between the upper and lower edges of the box represents the interquartile range (IQR); the lower and upper ends of the vertical line represent (Q1 – 1.5 × IQR) and (Q3 + 1.5 × IQR) as lower and upper limits. Any point outside these limits is considered an outlier. An outlier is represented as a black dot. BC, breast cancer; PET, positron emission tomography.

3.3. Secondary Outcomes

For Axilla‐BC, regarding missed finally included articles at 95% sensitivity for relevant articles, in the N = 500 and 1000 subsets, five articles were missed in each. In the N = 500 subset, this occurred across five trials, with one missed article per trial. In the N = 1000 subset, missed articles were distributed across three trials where two trials missed two articles each and one trial missed one article. For the N = 2000 subset in Axilla‐BC, eight articles were missed in total over six trials (5 trials missed 1, 1 trial missed 3). Therefore, for Axilla‐BC, 14 trials (considering all subsets) missed a total of 18 finally included articles at 95% sensitivity for relevant articles. For PET‐utility, overall, only one trial (corresponding to N = 500) missed one finally included article.

At 100% sensitivity for finally included articles, the median (range) of sensitivities for relevant articles identified for Axilla‐BC in the subsets of 500, 1000, and 2000 articles were 83% (56.8%–100.0%), 81.3% (61.4%–98.9%), and 96.3% (88.1%–100.0%), respectively. The corresponding values were lower in PET‐utility 78.6% (42.9%–100%), 71.2% (44.8%–96.6%), and 83.7% (62.1%–94.8%), respectively.

4. Discussions

In this study, we conducted a simulation‐based evaluation to compare the performance of Covidence's AI‐assisted Stage‐I screening function using two large‐scale SRs (Axilla‐BC and PET‐utility), each including over 7000 articles (typically > 5000 articles are considered as large [17]). These two SRs were selected to represent different purposes, to inform a guideline development and to advise a provincial steering committee in Ontario, Canada. We included random subsets of three sizes from the set of articles in these two SRs and executed multiple trials in these subsets to compare workload savings and time savings. We found that across all subsets and particularly for the N = 1000 subset, the workload and time savings at 95% sensitivity for relevant articles were significantly higher for Axilla‐BC than for PET‐utility.

The two SRs differed substantially across several key methodological and clinical dimensions, allowing us to examine how the performance of the AI‐assisted Stage‐I screening function varies under contrasting review characteristics. First, relevance and inclusion rates differed markedly: Axilla‐BC had almost three times the relevance rate and about half the inclusion rate of PET‐utility. Second, disease type varied considerably: Axilla‐BC focused exclusively on breast cancer, whereas PET‐utility spanned multiple cancer types, and three additional conditions (sarcoidosis, epilepsy and dementia). Notably, only one finally included article in the PET‐utility SR addressed a non‐cancer condition (dementia [18]). Third, the purpose and scope of the SRs differed: Axilla‐BC informed a nuanced CPG in axillary management of breast cancer with five research questions [13], whereas PET‐utility informed a high‐level report intended to summarize available evidence for decision‐making by a provincial committee [14]. Therefore, it could be reasonably stated that PET‐utility is extensive in terms of breadth of areas covered in cancer, but Axilla‐BC is extensive in depth and methodologically complex (including 5 research quesions), focusing on a single cancer site.

Considering the differences in terms of relevance rate, inclusion rate, disease type, and study purpose and scope, important performance distinctions in AI‐assisted screening emerge. In the Axilla‐BC SR, although workload and time savings were higher, finally included articles were missed in 20% of the trials at 95% sensitivity for relevance articles. Moreover, finally included articles tended to be identified later, occurring only after screening a median of at least 81% of relevant articles across all subsets. On the other hand, in the PET‐utility SR, only one trial missed one finally included article. Despite the span of this SR over multiple cancer types, only 1.5% of the trials missed finally included articles, and finally included articles were identified earlier, after screening a median of at least 71% of relevant articles. These findings suggest that Covidence's AI‐assisted screening function may perform more robustly for SRs that are broad in scope but less methodologically deep, compared with reviews that are narrow in scope but more complex and nuanced in inclusion criteria.

The underlying technical (i.e., machine learning‐based) reasons for these differences in performance cannot be determined from our study, because Covidence's AI‐assisted screening function is a proprietary black‐box system and technical details of the underlying learning algorithm are not publicly available. Therefore, we could not determine whether factors such as prevalence, topic heterogeneity, document similarity, or other machine learning‐related characteristics directly contributed to the observed differences. Future studies involving transparent algorithms or collaboration with tool developers may help clarify these mechanisms.

We evaluated Covidence's performance at two sensitivity thresholds (95% and 100%) for relevant articles. The 100% sensitivity threshold for relevant articles provides maximum protection against missing finally included articles. In contrast, evaluating performance at 95% sensitivity for relevant articles allows assessment of the potential impact of missing finally included articles (these could be present in the remaining 5% relevant articles) that could influence the findings and conclusions of the SR. While there are varied results at different sensitivity thresholds (ranging from 85% to 98%), all the finally included articles were found for the cancer‐related studies at 95% sensitivity as per a systematic review [10, 11]. Therefore, we chose the sensitivity threshold of 95% to determine whether this threshold allows for identifying all finally included articles. For AI‐assisted evidence synthesis, the recent Cochrane statements [6] emphasize human oversight and evaluation methodology based on the context. Our findings illustrate this context‐dependent tradeoff between efficiency and risk. For the Axilla‐BC SR, the 95% sensitivity threshold resulted in missing finally included articles in 20% of the trials, suggesting that this threshold may not provide sufficient reassurance in settings where missing eligible articles could affect final clinical commendations in the CPG. In such cases, a higher sensitivity threshold may be warranted despite lower workload savings. However, for the PET‐Utility SR, the 95% sensitivity threshold for relevant articles led to only one trial with a missed a finally included article, suggesting that this threshold may be acceptable while maintaining a very low risk of missing influential studies.

It is also noteworthy that, in parallel with the present study, we conducted a small prospective evaluation of Covidence's AI‐assisted Stage‐I screening applied to two updated evidence‐based CPGs, each with relatively modest sample sizes (1270 and 1734 records, respectively) [19]. The research questions for both CPGs were on intervention topics and comparatively straightforward. Consistent with our current findings, Covidence achieved substantial workload reductions at both 95% (one missed finally included reference per CPG) and 100% sensitivities for relevant articles. While preliminary, these observations further suggest that the AI‐assisted screening function may be more efficient and reliable in reviews with well‐defined questions and lower methodological complexity.

5. Limitations

Our study has several limitations. First, the simulation trials corresponding to the two SRs were executed sequentially (Axilla‐BC followed by PET‐utility). There could be internal changes at Covidence's end during the study time and our sequential execution cannot fully rule‐out time‐related confounding. However, no updates to machine‐learning modeling were announced by Covidence, during the study period, we conducted our trials (Dec 2024–May 2025). Therefore, the authors are unaware of any software version changes or platform modifications during that period and have reasonably assumed that the functionality remained consistent throughout the study period. Nevertheless, any changes made to Covidence (that affect the AI‐assistance function) after this period may need to be considered, should anyone seek to reproduce the findings of this study.

Second, as the subset of 2000 approximately required more than double the time than other two subsets (determined from our pilot trials), it was planned to restrict the number of trials to 10, assuming that this would not cause a significant loss of data and analytical quality. Furthermore, the extensive human involvement in the trial process also informed our overall scoping of this study as follows. We evaluated the performance of Covidence's AI‐assisted screening function using only two cancer‐related SRs and performed 70 trials for each SR using manageable subsets. Particularly, for the subset involving 2000 articles, we conducted 10 trials and that may have resulted in failure to determine significant differences between groups. Additional studies involving a larger number of SRs across diverse oncology topics are needed to further validate and generalize these findings. Third, we did not compare the performance of Covidence's AI‐assisted screening function to other platforms, such as EPPI‐Reviewer or DistillerSR, which have demonstrated favorable performance for Stage‐I screening in cancer‐related systematic reviews in a previous study [11]. Furthermore, our findings based on cancer‐related SRs may not readily be applicable to broader contexts, such as SRs that focus on non‐oncologic conditions. Finally, multiple statistical comparisons were not conducted across subset sizes, sensitivity thresholds, and outcome measures. We did not apply formal multiplicity adjustments because these analyses were pre‐planned (after conducting several pilot works), exploratory in nature, and intended to characterize Covidence's AI‐assisted screening function under different conditions. Consequently, the statistical findings should be interpreted with caution and in accordance with the context of our experiments.

6. Conclusions

In conclusion, Covidence's AI‐assisted Stage‐I screening demonstrated meaningful reduction in reviewer workload and screening time across two different cancer‐related systematic reviews. However, the magnitude of these reductions varied substantially between reviews as did the extent of the missed finally included articles. These findings suggest a possible association between the performance of Covidence's AI‐assisted screening function and the characteristics of an SR, including its topic, methodological complexity, relevance rate, and inclusion rate. Particularly in nuanced or methodologically complex oncology reviews, careful human oversight remains essential to ensure that all potentially eligible studies are identified, as missed articles may directly affect the quality and appropriateness of subsequent synthesized evidence. While our findings may not be directly generalizable to non‐oncology SRs or other types of evidence syntheses, the methodological insights gained from this study can inform future investigations of AI‐assisted screening tools in other disease contexts and review settings.

Author Contributions

Ashirbani Saha: conceptualization, methodology, software, data curation, investigation, validation, formal analysis, supervision, funding acquisition, visualization, project administration, resources, writing – original draft, writing – review and editing. Xiaomei Yao: conceptualization, investigation, funding acquisition, writing – original draft, methodology, validation, visualization, writing – review and editing, software, formal analysis, project administration, data curation, supervision, resources. Sharan Saravanan: data curation, investigation, validation, formal analysis, visualization, writing – original draft, writing – review and editing. Aditya Misra: data curation, investigation, validation, writing – original draft, writing ‐ review and editing. Ashley Low: project administration, validation, data curation, investigation, formal analysis, writing – review and editing. Samya Ali: data curation, investigation, validation, writing – review and editing. Mariam Abdelmalek: data curation, investigation, validation, writing – review and editing. Shakil Ahmed: data curation, investigation. Jonathan Sussman: writing – review and editing, conceptualization, methodology, funding acquisition, resources.

Ethics Statement

The authors have nothing to report.

Consent

The authors have nothing to report.

Conflicts of Interest

Prof. Xiaomei Yao is the member of the Cancer Innovation Editorial Board. To minimize bias, she was excluded from all editorial decision‐making related to the acceptance of this article for publication. The remaining authors declare no conflicts of interest. Covidence provided technical support during the simulation trials. The authors maintained full independence in study design, data analysis, and interpretation.

Saha A., Yao X., Saravanan S., et al., “Evaluating the Performance of Covidence's Artificial Intelligence‐based Screening Function in Cancer Systematic Reviews,” Cancer Innovation 5 (2026): e70077. 10.1002/cai2.70077.

Data Availability Statement

The data that support the findings of this study are available from the corresponding authors upon reasonable request.