Table 1.

Per-scenario overview of potential pitfalls and how to prevent these when using ASReview in a systematic review

Potential scenario		Pitfall	Remedy
①	Only a small (ie, manually feasible*) number of articles (with possibly a high proportion relevant) available for screening	Time wasted by considering AI-related choices, software training and no time saved by using AI	Do not use AI: conduct manual screening
②	Presence of duplicate articles in ASReview	Unequal weighing of labelled articles in AI-supported screening	Apply deduplication methods before using AI
③	Reviewer’s own opinion, expertise or mistakes influence(s) AI algorithm on article selection	Not all relevant articles are included, potentially introducing selection bias	Reviewer training in title and abstract screening Perform (partial) double screening and check inter-reviewer agreement
④	AI-supported screening is stopped before or a long time after all relevant articles are found	Not all relevant articles are included, potentially introducing selection bias, or time is wasted	Formulate a data-driven stopping criterion (ie, number of consecutive irrelevant articles)
⑤	AI-related choices not (completely) described	Irreproducible results, leading to a low-quality systematic review	Describe and substantiate the choices that are made
⑥	Study selection is not transparent	Irreproducible results (black box algorithm), leading to a low-quality systematic review	Publish open data (ie, extracted file with all decisions)

*What is considered manually feasible is highly context-dependent (ie, the intended workload and/or reviewers available).