Is AI-assisted active learning software able to reliably speed-up systematic literature reviews in rheumatology? A real-time comparison of AI-assisted and manual abstract selection

In evidence-based medicine, systematic literature reviews (SLRs) and meta-analyses are at the highest level of evidence, but the review process is very time-consuming.¹ Artificial intelligence (AI) can potentially improve many aspects of this process and tools have been developed for different stages of the systematic review process (search, screening, data extraction, appraisal/synthesis and analysis stage).² We evaluated the performance of the abstract screening tool ASReview, which can reduce screening time by 83%, on average, while identifying 95% of relevant articles.³ Previous publications applying this tool were overwhelmingly positive, but were tested in retrospective or simulated datasets. In ASReview, abstracts are screened, arranged by estimated relevance. Each subsequent abstract screened prompts the tool to rearrange the remaining abstracts, presenting the most relevant abstract next.⁴ In order to save time screening, it is necessary to predefine a stopping criterion to signify the screening has ‘finished’. We screened until at least 100 abstracts were screened and 50 consecutive abstracts were rejected. We performed an SLR using both manual screening and ASReview to investigate whether an increased dose of biological therapy in rheumatoid arthritis leads to better outcomes (search strategy in online supplemental file 1). All abstracts were screened by two reviewers: SAB and JAvdP. One hundred abstracts were randomly selected as manual screening training dataset (agreement 100% after consensus). The remaining abstracts were divided in two halves. Half A was screened by JAvdP using ASReview and manually by SAB, half B vice versa. Abstracts with opposing views were reassessed to reach consensus.

When using ASReviews, we first screened abstracts with the recommended AI classifier (Naïve Bayes).⁴ The output of this first model was presented to a more sophisticated model (neural networks) in order to ensure no abstracts were missed. We then screened the remaining abstracts until reaching the same stopping criterion, assuming we had identified all relevant abstracts.

Our search resulted in 4538 abstracts. After selecting 100 abstracts for the training dataset, 4438 articles were randomly divided in two halves. In half A, 318/2219 (14.3%) abstracts were screened by JAvdP using ASReview until the stopping criterion was reached. In 4 hours and 15 min, 83 relevant abstracts were retrieved. On second screening (with the second AI model), 1 relevant abstract was identified after screening 65 in 36 min. By manual screening, SAB identified 79 abstracts in 15 hours and 26 min. After consensus, 66 articles were included. Of these, 20 (30.3%) were not deemed relevant by the ASReview AI and would have been missed if no manual screening was applied, using manual screening as gold standard for abstract selection.

In half B, SAB reviewed 345/2219 (15.5%) abstracts using ASReview in 3 hours and 53 min until the stopping criterion was reached. In total, 78 relevant abstracts were retrieved. On second screening, 1 relevant abstract was identified after screening 72 additional abstracts in 48 min. By manual screening, JAvdP identified 92 relevant abstracts in 23 hours and 18 min. After consensus, 85 abstracts were included. Of these, 17 (20%) were not deemed relevant by the ASReview AI and would have been missed. In total, 151/4438 (3.4%) abstracts were identified by manual screening, of which 37 (24.5%) were missed by the ASReview tool. An overview of this selection process is presented in figure 1. In figure 1, the total number of abstracts is represented on the Y-axis, and the height of the columns represents the number of abstracts. AI-assisted screening was performed in 17.3% in half A and 18.8% of the time of manual abstract selection in half B, respectively. Figure 1 represents the amount of abstracts missed, when compared with the number of abstracts included. The full flow chart of the SLR can be found in online supplemental file 2.

Figure 1. (A, B) Abstract selection process. The blue bar represents the percentage of abstracts screened with the artificial intelligence (AI) compared with the total number of abstracts in half A and half B (2219 each). The time in minutes of manual selection versus screened with the AI model is presented. The orange bar represents the number of abstracts selected after consensus (66 and 85, respectively). The yellow bar represents the percentage of abstracts missed by the AI model. The systematic literature reviews results will be presented separately.

AI-assisted review techniques could greatly improve the SLR process. However, in our analysis, applying such a technique resulted in missing nearly 25% of relevant abstracts. These findings cannot be extrapolated to other tools than ASReview. Our research question was complex, including many types of drugs in both trials and observational databases. With a more specific research question, the model might be more accurate. Our results could be improved by choosing a stricter stopping criterion, at the cost of an increased number of abstracts to be screened. In conclusion, AI-assisted review techniques seem very promising, but based on our prospective comparison to manual abstract selection, further improvements are warranted before this technique can be reliably applied to study complex research questions in rheumatology due to the high percentage of relevant abstracts missed.