Table 3.

List and evaluation criteria of all systematic review (SR) tasks and systematic review automation (SRA) tools used.

SR task No.	SR task	SRA tool used	Evaluation criteria
1	Formulate the question	N/Aa	N/A
2	Check for similar reviews	N/A	N/A
3	Write the protocol	N/A	N/A
4	Design the systematic search	N/A	N/A
5	Run the systematic search	Polyglot Search Translator [8]	Completed by one participant; the number of different types of errors were counted.
6	Deduplicate the search results	Deduplicator [9]	Completed by one participant; deduplicated EndNote libraries were compared to a deduplicated reference standard data set.
7	Screen the titles and abstracts	SRA-Helperb [20]	Completed by two participants; EndNote libraries of the included and excluded studies were compared. A wrongfully excluded study was considered an error.
8	Find the full text	EndNote, SRA-Helper [20], and SARAc [21]	Completed by one participant; the number of references ordered through the library was compared.
9	Screen the full text	SRA-Helper [20]	Completed by two participants; EndNote libraries of the included and excluded studies were compared.
10	Conduct a citation analysis	N/A	N/A
11	Extract the data	N/A	N/A
12	Assess the risk of bias	RobotReviewer [12]	Completed by two participants; the risk-of-bias assessments were compared to a reference standard created by two experienced systematic reviewers external to the two teams.
13	Synthesize the data	N/A	N/A
14	Rerun the systematic search	N/A	N/A
15	Conduct a meta-analysis	N/A	N/A
16	Write the results	N/A	N/A

^aN/A: not applicable; this task did not have any relevant SRA tools.

^bSRA-Helper: Systematic Review Accelerator Helper.

^cSARA: System for Automatically Requesting Articles.