Skip to main content
. 2022 May 2;10(5):e33219. doi: 10.2196/33219

Table 2.

The criteria for each selected feature, as well as the rationale.

Classification and variable name and coding Feature from Rationale (if added by authors)
Retrieval

Database search 1—literature search through APIa Integration with a database; 0—no method for retrieving studies directly from a database Kohl et al [12], Marshall et al [11] b

Reference importing 1—import of references as RISc files or other file types; 0—references have to be entered manually Harrison et al [10], Van der Mierden et al [9]

Manual addition 1—add a reference by entering study metadata; 0—no method for adding individual references and gray literature Added by the authors Ability to add expert additions is called for by the PRISMAd 2020 guidelines and checklist [14]

Attaching full-text PDFs 1—ability to import or upload full-text PDFs associated with each study under review; 0—no method for importing full-text PDFs in the screening process Harrison et al [10], Van der Mierden et al [9]

Automated full-text retrieval 1—ability to fetch some or all full texts via API or other nonmanual method; 0—full texts must be uploaded manually, or full-text upload not supported Added by the authors Full texts are required for content extraction, and manual upload represents a major time investment by the user
Appraisal

Title/abstract screening 1—inclusion and exclusion by title and abstract only; 0—no system for inclusion and exclusion of references by title and abstract Harrison et al [10], Van der Mierden et al [9]

Full-text screening 1—a distinct full-text screening phase; 0—there is no full-text screening phase Harrison et al [10], Van der Mierden et al [9]

Dual screening and adjudication 1—choice for single or double screening and a method for resolving conflicts; 0—no ability to configure screening mode or no ability to resolve conflicts Harrison et al [10], Van der Mierden et al [9]

Keyword highlighting 1—abstract keywords are highlighted. Keywords can be user or AIe-determined; 0—No keyword highlighting is possible Harrison et al [10], Van der Mierden et al [9]

Machine learning/automation (screening) 1—has a form of machine learning or automation of the screening process; 0—does not support any form of machine learning or automation of the screening process Added by the authors Automated screening has been called for by the scientific community [15]

Deduplication of references 1—automatically identifies duplicate references or marks potential duplicates for manual review; 0—has no mechanism for deduplication Harrison et al [10], Kohl et al [12]
Extraction

Tagging references 1—ability to attach tags that reflect the content of underlying studies to specific references; 0—no means for attaching content-related tags to references Van der Mierden et al [9], Kohl et al [12]

Data extraction 1—facilitates extraction and storage of quantitative data into a form or template; 0—does not permit extraction and storage or quantitative data Harrison et al [10], Kohl et al [12], Marshall et al [11]

Dual extraction 1—ability for 2 independent reviewers to collect on each study and for a third person to adjudicate differences; 0—no ability to have independent extraction and adjudication Added by the authors Dual extraction improves the accuracy of data gathering [16]

Risk of bias 1—supports critical appraisal of studies through risk of bias assessments; 0—no built-in features or templates to assess risk of bias Kohl et al [12]
Documentation/output

Flow diagram creation 1—automated or semiautomated creation of PRISMA flow diagrams; 0—the tool cannot automatically provide a flow diagram meeting the PRISMA criteria Van der Mierden et al [9]

Manuscript writing 1—ability to write or edit a report or manuscript; 0—no ability to write or edit a report or manuscript Marshall et al [11]

Citation management 1—ability to insert citations based on stored study metadata into a text editor; 0—no ability to insert citations into a document Added by the authors The ability to add and manage citations is necessary to document the source of review data

Data visualizations 1—generation of figures or tables to assist with data presentation; 0—no built-in way to generate figures or tables Kohl et al [12]

Export 1—supports export of references, study metadata, or collected data; 0—has no export feature Harrison et al [10], Van der Mierden et al [9]
Admin

Protocol 1—supports protocol development or filling in a research question template; 0—no protocol development or templates Kohl et al [12], Marshall et al [11]

Distinct user roles 1—distinct user roles and permissions; 0—no distinct roles; everybody has the same role and rights in the project Harrison et al [10], Van der Mierden et al [9], Marshall et al [11]

Activity monitoring 1—software monitors and displays progress through the project; 0—there is no way to determine overall progress of the project (eg, % completed) Harrison et al [10], Van der Mierden et al [9]

Comments or chat 1—ability to leave comments or notes on studies; 0—it is not possible to attach comments to references Van der Mierden et al [9]

Training 1—there are publicly available web-based tutorials, help pages, training videos, or forums maintained by the software provider; 0—there are no accessible tutorials or training materials maintained by the software provider Harrison et al [10], Marshall et al {11]

Customer support 1—customer support, such as support contact information, is provided on request; 0—customer support is not clearly available Van der Mierden et al [9]
Access and support

Pricing (free to use) 1—a free version is available for users; 0—the tool must be purchased, or free or trial accounts have severe limitations that can compromise the systematic review Harrison et al [10], Van der Mierden et al [9], Marshall et al [11]

Living/updatable 1—new records can be added after a project has been completed; 0—new records cannot be added after a project has been completed Added by the authors Living systematic review has been called for as a novel paradigm solving the main limitation of systematic review [17]

Public outputs 1—web-based visualizations or writing can be made publicly visible; 0—review data and outputs cannot be made publicly visible Added by the authors Web-based availability of systematic review outputs is important for transparency and replicability of research [18]

User collaboration 1—multiple users can work simultaneously on 1 review; 0—it is not possible for multiple users to work at the same time on the same project, independently Harrison et al [10], Van der Mierden et al [9], Marshall et al [11]

aAPI: application programming interface.

bRationale only provided for features added in this review; all other features were drawn from existing feature analyses of Systematic Review Software Tools.

cRIS: Research Information System.

dPRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

eAI: artificial intelligence.