Automated tools in systematic reviews: Current trends

INTRODUCTION

Systematic reviews and meta-analyses (SRMA) offer a pooled synthesis of available literature to inform clinical decision-making. The traditional methods of conducting systematic reviews require significant time and human resources and are also vulnerable to human-related error.[1] To overcome these challenges, the integration of automated tools has emerged as an advancement which enhances both the efficiency and quality of SRMA.[2,3] Automated tools assist or replace manual tasks with literature searches, study screening, data extraction, quality assessment, and statistical synthesis. Various automated platforms, such as Covidence and Rayyan, are increasingly being adopted to facilitate the review process. Automated tools facilitate the real-time synthesis of data, including forest plots, heterogeneity analysis, and subgroup and sensitivity testing.[4] In recent years, globally, researchers have adopted automated tools.[5] However, the degree to which these tools have been incorporated into systematic reviews published in the Indian Journal of Anaesthesia (IJA) is underexplored. This article explores the various automated tools employed during the review process, including screening, data extraction, and analysis.

METHODS

We conducted a cross-sectional study aimed at identifying the use of various automated tools in systematic reviews with or without meta-analysis published in the IJA from 1 July 2024 to 30 June 2025. All related issues of the IJA were manually screened through the journal’s website independently by two authors (AL and RGM), with disagreements resolved by consensus. All articles explicitly identified as systematic reviews in the title or abstract were included. The following data were extracted using a pre-designed data extraction form: Article title, type of systematic review (with or without meta-analysis), and automated tools used at any stages of the review process (e.g., literature screening, data extraction, statistical analysis, risk of bias assessment, citation management or any others). The data were analysed descriptively. The type of automated tool and stages of review where tools were applied were summarised.

RESULTS

All 22 included systematic reviews utilised at least one automated tool during at least one step of the review process [Table 1].[6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27] Fifty per cent of the included studies (11/22) employed an automated tool during literature screening using either EndNote, Rayyan, or Covidence.[6,8,9,11,12,18,19,20,24,25,26] One study did not mention the name of the automated tool, although the authors had mentioned the use of an automated tool.[12] None of the included studies used any automated tools for Data Extraction. A total of 18% (4/22) of studies used automated tools for risk of bias (ROB) assessment.[7,11,14,23] One used Review Manager (RevMan 5.4),[7] while the other three used the ROBVIS tool for visualisation of the ROB chart.[11,14,23] All included studies utilised an automated tool for statistics or meta-analyses, specifically R software [22.7% (5/22)],[6,8,11,17,23] RevMan [63.6% (14/22)],[7,9,10,12,13,15,16,18,19,20,21,24,26,27] STATA [9% (2/22)],[14,25] and Jamovi and Comprehensive Meta-Analysis (CMA) software [4.5% (1/22)].[22] Though 72% (16/22)[6,8,9,10,12,13,14,15,16,17,19,20,21,22,23,25] of studies have used GRADE to summarise the certainty of evidence, only 31% (7/22) of studies[10,14,15,19,20,23,27] have explicitly mentioned the use of GRADEpro GDT software. None of the authors of any studies made a declaration of using automated tools for manuscript writing or editing.

Table 1.

Details of automated tools used in published studies

Stage of Review	Number of Studies using Automated Tools (n=22)	Automated Tools
Literature Screening, n (%)	11 (50%)	EndNote, Rayyan, Covidence
Data Extraction, n (%)	0 (0%)	-
Risk of Bias Assessment, n (%)	4 (18%)	RevMan 5.4, ROBVIS
Statistical Analysis/Meta-analysis, n (%)	22 (100%)	RevMan, R software, Stata, Jamovi, CMA
GRADE Assessment, n (%)	7 (32%)	GRADE-pro
Manuscript drafting/Writing, n (%)	0/(0%)	-

GRADE=Grading of Recommendations Assessment, Development and Evaluation, ROBVIS=Risk Of Bias Visualisation, CMA=Comprehensive Meta-Analysis

DISCUSSION

Various automated tools are increasingly used in systematic reviews to enhance efficiency and reproducibility. [Table 2]. Our study findings demonstrated that while automated tools are being integrated into the systematic review process in the IJA, their use is primarily confined to literature screening and statistical analysis. Many critical review steps, such as data extraction and ROB assessment, are still based on manual methods. The use of automated tools, such as Rayyan and EndNote, for duplicate removal and literature screening indicates an awareness of available technologies. The minimal use of tools for ROB visualisation and the complete lack of automated data extraction suggest opportunities for greater integration of advanced automated tools. GRADEpro was used in nearly one-third of the studies, indicating a growing attention to the quality and certainty of evidence.

Table 2.

Commonly used automated tools for systematic reviews

Automated tool	Basic details
Rayyan	Link: https://www.rayyan.ai/ Used for: •Literature screening •Data for PRISMA flowchart preparation Advantage: •Free version for early researchers, •Real-time blinded screening. •The mobile app is also available. •Limitations: Features of Data extraction and ROB assessment are not available.
Covidence	Link: https://www.covidence.org/ Used for: •Literature screening •Data for PRISMA flowchart preparation •Data extraction •Risk of Bias assessment Advantages: •Real-time blinded screening. •Features of Data extraction and ROB assessment are available. •Customisable data extraction forms •Limitation: The Free version has limited features.
RevMan	Link: https://revman.cochrane.org/info Used for: •Calculation and graphical display of meta-analyses (e.g., forest plots, risk of bias, funnel plots). Advantages: •Designed for Cochrane-style reviews with built-in templates and tools •RevMan Web supports cloud-based editing and collaboration •Easy data transfer from screening tools like Covidence Limitation: •Lacks advanced statistical modelling or integration with statistical software like R or Stata. •Does not include built-in reference management •RevMan Web requires an Internet connection •Paid software
ROBVIS	Link: https://www.riskofbias.info/welcome/robvis-visualization-tool used for: •Generates Traffic Light Plots for risk of bias assessments Advantages: •Free •Web-based, no installation Limitation: •Not for ROB assessment •Manual data input •Web-based, need Internet access
GRADEpro GDT	Link: https://www.gradepro.org/ Used for: •Create, edit, and export GRADE summary of findings Advantages: •Collaborative work in real-time on evidence tables. •Supports integration with RevMan •Tables and reports can be exported to Word, Excel, and PDFs. •No installation is needed •Free Limitation: •Web-based, need Internet access
STATA	Link: https://www.stata.com/ Used for: •Data management, meta-analysis, Forest plot, Heterogeneity test, etc. Advantages: •Advanced statistical tools, e.g., Metareg •Support complex analysis •Can import data from Excel easily Limitation: •Paid software •No automated data extraction

PRISMA=Preferred Reporting Items for Systematic reviews and Meta-Analyses, GRADE=Grading of Recommendations Assessment, Development and Evaluation, ROBVIS=Risk of Bias VISualization, ROB=Risk of bias, STATA=Statistical software for data science

The commonly used literature screening automated tools are Rayyan, Covidence, and EndNote. These tools assist with reference management, duplicate removal, and abstract screening. Rayyan has become increasingly popular among reviewers due to its ease of collaboration and real-time blinding.[28,29] Covidence is endorsed by Cochrane and is designed to support the systematic review process. It helps in article screening, duplicate removal, full-text review, data extraction, and ROB assessment. Data can be easily imported from EndNote, Zotero, or any other search engine. It provides real-time blinding and conflict resolution.[30] EndNote, a paid citation manager, is often used in combination with other automated tools.[31] A study by Harrison et al.[32] suggested that Covidence and Rayyan provide a user-friendly experience for article screening. These two automated tools consistently performed well, demonstrating good alignment with researchers’ requirements.

Data extraction is a time-consuming and error-prone step in systematic reviews. Automated tools, including RobotReviewer, are available to assist with data extraction. The lack of awareness or training, perceived unreliability, and limited accessibility of tools could be possible reasons for not using these tools.[33]

RevMan can provide the ROB assessment; however, its application is still manual and subjective. RobotReviewer also provides the ROB assessment by identifying relevant text from full-text articles and assigning judgments. Despite the availability of such tools, their adoption in included studies was minimal. A study by Marshall et al.[34] showed that the performance of RobotReviewer was comparable to that of human reviewers in domains such as random sequence generation and allocation concealment; however, it was less reliable in selective reporting. Such inconsistencies may discourage researchers from automated ROB assessment.

The RevMan, developed by the Cochrane Collaboration, provides an interface and integrated meta-analytic functions, including heterogeneity testing. R software is relatively complex, which is increasingly being used for meta-analysis.

GRADEpro GDT facilitates the generation of evidence-based recommendations based on ROB, consistency, indirectness, precision, and publication bias. The use of GRADE tools in only two-thirds of included studies suggests that certainty assessment is still not consistently implemented, which may impact the practical relevance of the results.

However, this study has a few limitations. The study was limited to one journal and to one year duration, which limits its generalisability. We did not contact the authors for disclosure of automated tools. The possibility of underreporting the use of automated tools must be considered.

CONCLUSION

The use of automated tools in systematic reviews published in the IJA has shown encouraging trends. Initiatives such as training are needed to enable researchers to work effectively with the growing range of automation tools. We need a large-scale audit across multiple journals and years to explore the use of automated tools in systematic reviews.

Study data availability

De-identified data may be requested with reasonable justification from the authors (email to the corresponding author) and shall be shared upon request.

Disclosure of use of artificial intelligence (AI)-assistive or generative tools

The AI tools or language models (LLM) have not been utilised in the manuscript, except that software has been used for grammar corrections.

Authors contributions

IM: Concept, design, Literature search, Manuscript preparation. AL: Literature search, Data analyses, Manuscript editing. RGM: Literature search, Data acquisition, Manuscript preparation. RG: Concept, Design, Manuscript editing, review and approval.

Supplementary material

None.

Declaration of Use of Permitted Tools

Not applicable.

Presentation at conferences/CMEs and abstract publication

Nil.

Conflicts of interest

Dr. Indubala Maurya and Dr. Rakesh Garg, who are one of the co-authors of this manuscript, are editors of this journal. They were not involved in any decision-making process, and an independent editor handled this manuscript. The authors declare that they have no other conflicts of interest.

Funding Statement

Nil.