Table 1.
Preliminary list of considerations constructed in stage 1 and the corresponding item number for the stage 2 survey.
| no. | consideration | corresponding itema |
|---|---|---|
| 1 | Do you consider the current strength of evidence in favour for the claim to be weak (as for example quantified by a Bayes factor, a very wide CI, or a p-value close to the typical alpha level of 0.05 combined with a very large sample size)? | Q7 item 12 |
| 2 | Given the current state of investigation of this claim in the literature, how certain are you that the claim is true? Please motivate your answer. | Q5 item 5 |
| 3 | Is the claim theoretically important? If yes, please elaborate. | Q5 item 4 |
| 4 | Do you perceive this claim to have relevant implications, for instance in practice, policy or clinical work? If yes, please elaborate. | Q5 item 3 |
| 5 | Please describe the design of the original study. | Q7 item 2–5 |
| 6 | Enter the sample size. | Q7 item 1 |
| 7 | Who was the sample (for example, what were inclusion and exclusion criteria)? | Q7 item 2 |
| 8 | How was the main outcome measured? | Q7 item 19 |
| 9.1 | Do you consider the outcome measure to be valid? Please motivate your answer. | Q7 item 9 |
| 9.2 | Do you consider the outcome measure to be reliable? Please motivate your answer. | Q7 item 10 |
| 9.3 | Do you consider the outcome measure to be biased? Please motivate your answer. | Q7 item 11 |
| 10 | Do you consider the operationalization appropriate (i.e. are the methods fitted to answer the broader research question that was posed)? | Q7 item 20 |
| 11 | Please describe the analysis plan and performed analysis. | Q7 item 13, 14, 17 |
| 12 | Please enter the observed effect size. | Q7 item 6,7 |
| 13 | Given the sample characteristics, was the sample a good representation of the population? In other words, do the results generalize to the population of interest? | Q7 item 8 |
| 14 | Is the interpretation of the current claim limited by potential confounds? If yes, please describe. | Q7 item 21 |
| 15.1 | Given the original study set-up, is replication readily feasible? | Q9 item 2 |
| 15.2 | Can this study be replicated by generally equipped labs, or are more specific experimental set-ups necessary (e.g. an eye-tracking machine, an fMRI-scanner, a sound-proof booth, etc.)? | Q9 item 1 |
| 16 | How could a replication overcome the issues you raised above? Please also specify the type of replication you intend to run (i.e. close or conceptual). |
aItem numbers refer to the presentation in the electronic supplementary material.