Table 2.
Classification and variable name and coding | Feature from | Rationale (if added by authors) | ||
Retrieval | ||||
|
Database search | 1—literature search through APIa Integration with a database; 0—no method for retrieving studies directly from a database | Kohl et al [12], Marshall et al [11] | —b |
|
Reference importing | 1—import of references as RISc files or other file types; 0—references have to be entered manually | Harrison et al [10], Van der Mierden et al [9] | — |
|
Manual addition | 1—add a reference by entering study metadata; 0—no method for adding individual references and gray literature | Added by the authors | Ability to add expert additions is called for by the PRISMAd 2020 guidelines and checklist [14] |
|
Attaching full-text PDFs | 1—ability to import or upload full-text PDFs associated with each study under review; 0—no method for importing full-text PDFs in the screening process | Harrison et al [10], Van der Mierden et al [9] | — |
|
Automated full-text retrieval | 1—ability to fetch some or all full texts via API or other nonmanual method; 0—full texts must be uploaded manually, or full-text upload not supported | Added by the authors | Full texts are required for content extraction, and manual upload represents a major time investment by the user |
Appraisal | ||||
|
Title/abstract screening | 1—inclusion and exclusion by title and abstract only; 0—no system for inclusion and exclusion of references by title and abstract | Harrison et al [10], Van der Mierden et al [9] | — |
|
Full-text screening | 1—a distinct full-text screening phase; 0—there is no full-text screening phase | Harrison et al [10], Van der Mierden et al [9] | — |
|
Dual screening and adjudication | 1—choice for single or double screening and a method for resolving conflicts; 0—no ability to configure screening mode or no ability to resolve conflicts | Harrison et al [10], Van der Mierden et al [9] | — |
|
Keyword highlighting | 1—abstract keywords are highlighted. Keywords can be user or AIe-determined; 0—No keyword highlighting is possible | Harrison et al [10], Van der Mierden et al [9] | — |
|
Machine learning/automation (screening) | 1—has a form of machine learning or automation of the screening process; 0—does not support any form of machine learning or automation of the screening process | Added by the authors | Automated screening has been called for by the scientific community [15] |
|
Deduplication of references | 1—automatically identifies duplicate references or marks potential duplicates for manual review; 0—has no mechanism for deduplication | Harrison et al [10], Kohl et al [12] | — |
Extraction | ||||
|
Tagging references | 1—ability to attach tags that reflect the content of underlying studies to specific references; 0—no means for attaching content-related tags to references | Van der Mierden et al [9], Kohl et al [12] | — |
|
Data extraction | 1—facilitates extraction and storage of quantitative data into a form or template; 0—does not permit extraction and storage or quantitative data | Harrison et al [10], Kohl et al [12], Marshall et al [11] | — |
|
Dual extraction | 1—ability for 2 independent reviewers to collect on each study and for a third person to adjudicate differences; 0—no ability to have independent extraction and adjudication | Added by the authors | Dual extraction improves the accuracy of data gathering [16] |
|
Risk of bias | 1—supports critical appraisal of studies through risk of bias assessments; 0—no built-in features or templates to assess risk of bias | Kohl et al [12] | — |
Documentation/output | ||||
|
Flow diagram creation | 1—automated or semiautomated creation of PRISMA flow diagrams; 0—the tool cannot automatically provide a flow diagram meeting the PRISMA criteria | Van der Mierden et al [9] | — |
|
Manuscript writing | 1—ability to write or edit a report or manuscript; 0—no ability to write or edit a report or manuscript | Marshall et al [11] | — |
|
Citation management | 1—ability to insert citations based on stored study metadata into a text editor; 0—no ability to insert citations into a document | Added by the authors | The ability to add and manage citations is necessary to document the source of review data |
|
Data visualizations | 1—generation of figures or tables to assist with data presentation; 0—no built-in way to generate figures or tables | Kohl et al [12] | — |
|
Export | 1—supports export of references, study metadata, or collected data; 0—has no export feature | Harrison et al [10], Van der Mierden et al [9] | — |
Admin | ||||
|
Protocol | 1—supports protocol development or filling in a research question template; 0—no protocol development or templates | Kohl et al [12], Marshall et al [11] | — |
|
Distinct user roles | 1—distinct user roles and permissions; 0—no distinct roles; everybody has the same role and rights in the project | Harrison et al [10], Van der Mierden et al [9], Marshall et al [11] | — |
|
Activity monitoring | 1—software monitors and displays progress through the project; 0—there is no way to determine overall progress of the project (eg, % completed) | Harrison et al [10], Van der Mierden et al [9] | — |
|
Comments or chat | 1—ability to leave comments or notes on studies; 0—it is not possible to attach comments to references | Van der Mierden et al [9] | — |
|
Training | 1—there are publicly available web-based tutorials, help pages, training videos, or forums maintained by the software provider; 0—there are no accessible tutorials or training materials maintained by the software provider | Harrison et al [10], Marshall et al {11] | — |
|
Customer support | 1—customer support, such as support contact information, is provided on request; 0—customer support is not clearly available | Van der Mierden et al [9] | — |
Access and support | ||||
|
Pricing (free to use) | 1—a free version is available for users; 0—the tool must be purchased, or free or trial accounts have severe limitations that can compromise the systematic review | Harrison et al [10], Van der Mierden et al [9], Marshall et al [11] | — |
|
Living/updatable | 1—new records can be added after a project has been completed; 0—new records cannot be added after a project has been completed | Added by the authors | Living systematic review has been called for as a novel paradigm solving the main limitation of systematic review [17] |
|
Public outputs | 1—web-based visualizations or writing can be made publicly visible; 0—review data and outputs cannot be made publicly visible | Added by the authors | Web-based availability of systematic review outputs is important for transparency and replicability of research [18] |
|
User collaboration | 1—multiple users can work simultaneously on 1 review; 0—it is not possible for multiple users to work at the same time on the same project, independently | Harrison et al [10], Van der Mierden et al [9], Marshall et al [11] | — |
aAPI: application programming interface.
bRationale only provided for features added in this review; all other features were drawn from existing feature analyses of Systematic Review Software Tools.
cRIS: Research Information System.
dPRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
eAI: artificial intelligence.