Table 4.

Quantitative results of stage 3 checklist development over the course of the three rounds. Included items are highlighted in bold and revisions are indicated in italic.

item	round 1		round 2		round 3		decision
	Mdn	IQR	Mdn	IQR	Mdn	IQR
the relevance of the original study for your current line of research or the field you work in	7	2					include
your involvement in the line of research that the replication target is concerned with (e.g. self-replication, planning to build on the study in the future)	6	3					revise
the degree of involvement you have in previous or upcoming projects related to the replication target (e.g. self-replication, planning to build on the study in the future)			6	3.5			revise
your personal stakes in the replication target’s results (e.g. self-replication, financial stakes or other potential conflicts of interest, planning to build on the replication target results in future research, etc.)					7.5	3	no consensus
the current strength of evidence in favour of the original claim (e.g. a high/low Bayes factor, a wide/narrow confidence interval, a high/low p-value)	7	1					include
your personal belief about the truthfulness of the original claim (e.g. consensus in findings, replication attempts)	5	2					exclude
your expectations about whether the original claim would replicate or not	5	2					exclude
the importance of the original study for research (e.g. often/rarely cited, under/over-studied, published in high/low impact journal)	7	1.5					include
the theoretical relevance of the original claim	8	2					include
implications of the original claim (e.g. for practice, policy or clinical work)	8	2					include
the clarity and replicability of the original protocol (e.g. completeness and clarity of the methodological description, accessibility of the materials)	6	4					re-evaluate
—			4	4.25			revise
the (un)clarity and (un)replicability of the original protocol (e.g. completeness and clarity of the methodological description, accessibility of the materials)					7.5	2	include
the sample size of the original study (too small or too large)	7	2					include
flaws of the original design (e.g. in- an exclusion criteria, potential confounds)	8	1.5					include
operationalization of the original study’s measures (e.g. validity, reliability, and bias)	7	3					re-evaluate
—			7	2.25			revise
operationalization of the original study’s measures (e.g. validity, reliability, and bias) and how this impacts the credibility of the original study					7	1.25	include
concerns that questionable research practices have been employed (e.g. presence/absence of preregistration, potential of p-hacking or HARKing)	7	2					include
generalizability of the original finding (e.g. cultural and temporal context, representativeness of the sample)	7	2					include
the resources available to you for replicating the original study (e.g. funding, time, equipment, study materials, or data)	8	2					include
the adaptability of the original study design (e.g. mode of data collection, whether the study can be translated into other languages, contexts)	6	2.5					re-evaluate
—			6	2.25			revise
the adaptability of the original study design (e.g. whether data is collected online or on-site, whether the study can be translated into other languages or applied to different contexts, etc.)					6.5	1	exclude
your previous experience and expertise with regards to the original study	5	4					revise
you (i.e. all replicating authors) previous experience and expertise with regards to the original study			5.5	3			revise
your (i.e. the replicating team as a whole) presence or absence of previous experience or expertise on the original study as a practical concern					7	2	include
educational value of conducting the replication study (e.g. for a thesis or student project)	5	3.5					re-evaluate
—			3	4			re-evaluate
—					5	1.5	exclude

Note: Re-evaluate means that participants received qualitative feedback and were asked to rate the same item again.