Table 1.
Reproducibility indicators
| Reproducibility indicator | Role in producing transparent and reproducible science | Original sample (n = 300) |
|---|---|---|
| Articles | ||
| Article accessibility: articles were assessed for open accessibility, paywall access, or unable to access full text | Ease of access to publications enables interdisciplinary research by removing access barriers. Full-text access allows for validation through reproduction. | All (n = 300) |
| Funding | ||
| Funding statement: presence of funding sources of the study | Funding provides researchers the ability to create new experiments and tangibly investigate their ideas. However, funding sources can play a role in influencing how researchers conduct and report their study (e.g., scientific bias), which necessitates its disclosure. | All included publications† (n = 294) |
| Conflict of interest (COI) | ||
| COI statement: presence of conflict of interest statement | Conflict of interest conveys the authors’ potential associations that may affect the experimental design, methods, and analyses of the outcomes. Thus, full disclosure of possible conflicts allows for unbiased presentation of their study. | All included publications† (n = 294) |
| Data | ||
| Data statement: presence of a data availability statement, retrieval method, comprehensibility, and content | Raw data availability facilitates independent verification of research publications. It can improve accountability of outcomes reported and integrity of the research published. | Empirical publications‡ (n = 195) |
| Pre-registration | ||
| Pre-registration statement: presence of statement indicating registration, retrieval method, accessibility, and contents (hypothesis, methods, analysis plan) | Pre-registration explicitly reports aspects of the study design prior to the commencement of the research. Pre-registration functions as a way to limit selective reporting of results and prevents publication biases and P-hacking. | Empirical publications‡ (n = 195) |
| Protocols | ||
| Protocol statement: assessed for statement indicating protocol availability, and if available, what aspects of the study are available (hypothesis, methods, analysis plan) | Reproducibility of a study is dependent on the accessibility of the protocol. A protocol is a highly detailed document that contains all aspects of the experimental design which provides a step by step guide in conducting the study. | Empirical publications‡ (n = 195) |
| Analysis scripts | ||
| Analysis scripts statement: presence of analysis script availability statement, retrieval method, and accessibility | Analysis scripts are used to analyze data obtained in a study through software programs such as R, Python, and MatLab. Analysis scripts provide step by step instructions to reproduce statistical results. | Empirical publications‡ (n = 195) |
| Replication | ||
| Replication statement: Presence of statement indication a replication study. | Replication studies provide validation to previously done publications by determining whether similar outcomes can be acquired. | Empirical studies‡ (n = 195) |
| Materials | ||
| Materials statement: presence of a materials availability statement, retrieval method, and accessibility | Materials are the tools used to conduct the study. Lack of materials specification impedes the ability to reproduce a study. | Empirical publications¶ (n = 191) |
The indicators measured for the articles varied depending on its study type. More details about extraction and coding are available here https://osf.io/x24n3/
†Excludes publications that have no empirical data
‡Excludes case studies and case series
¶Excludes meta-analyses and systematic reviews, which materials may not be relevant, in addition to ‡