Table 3.
Stage 2 survey questions with number of available responses, median rating (1 = not at all important, 9 = very important), and corresponding interquartile range.
| question | N | median | IQR |
|---|---|---|---|
| Please indicate to what extent you considered the following pieces of information when scrutinizing the potential replication target: | |||
| — Whether the finding has been investigated sufficiently or not. | 119 | 8 | 3 |
| — Whether the citation count of the study was high or low. | 119 | 4 | 5 |
| — Whether the study has relevant implications, for instance in practice, policy, or clinical work, or not. | 120 | 7 | 3 |
| — Whether the finding has a strong connection with theory or not. | 120 | 7 | 3 |
| — Whether the finding was unexpected (e.g. ‘counterintuitive’, ‘surprising’), or in line with what can be expected. | 119 | 6 | 4 |
| Please indicate how important the following specific characteristics of the original study were for you when choosing your replication target: | |||
| — The total sample size. | 115 | 6 | 4 |
| — Handling of inclusion and exclusion criteria. | 115 | 4 | 5 |
| — Blinding procedures (e.g. blinding of participants, experimenters, analysers). | 117 | 2 | 5 |
| — Sampling procedures (e.g. stratified random sampling, snowball sampling, convenience sampling, etc.). | 115 | 4 | 4.5 |
| — How participants were assigned to conditions (e.g. randomly, single/double blind, etc.). | 116 | 3 | 5 |
| — Statistical power to detect the effect sizes of interest. | 116 | 6 | 4.25 |
| — The size of the effect size. | 119 | 6 | 4 |
| — Generalizability of the sample. | 116 | 5 | 4 |
| — Validity of the outcome measures. | 116 | 6 | 3 |
| — Reliability of the outcome measures. | 114 | 6 | 4 |
| — Potential bias of the outcome measures. | 115 | 5 | 5 |
| — The strength of evidence (measured by reported p-value, confidence interval, Bayes Factor, etc.). | 117 | 7 | 2 |
| — Missing data handling. | 114 | 3 | 4 |
| — Whether the finding was based on within-subject measurements or between-subject measurements. | 116 | 3 | 4 |
| — Open access to underlying empirical data that were analysed. | 117 | 3 | 4 |
| — Whether the study has been preregistered. | 118 | 2 | 4 |
| — Whether the finding was predicted a priori or discovered during data exploration. | 114 | 5 | 5 |
| — Whether there are statistical errors in the results reported (e.g. the degrees of freedom do not correspond to the other reported statistics, the total sample size does not equal the sum of the group sample sizes, etc.). | 114 | 4 | 5 |
| — How the main outcome was measured. | 116 | 6 | 4 |
| — Whether the operationalizations were appropriate (i.e. the methods were fit to answer the broader research question that was posed). | 115 | 5 | 4 |
| — Whether interpretation of the results was limited by potential confounds or not. | 114 | 6 | 4 |
| Please indicate how important the following pieces of information were for you when judging the feasibility of your replication study: | |||
| — Whether the study could be replicated by a laboratory without specialized equipment (e.g. an eye-tracker, a sound-proof lab, an MRI-scanner). | 119 | 7 | 5 |
| — Whether the study concerned a hard-to-collect sample. | 118 | 7 | 5 |