Table 2.

Cost-benefit analysis of the credibility-enhancing methods. Note: for the single-site study, we calculated with eight consensus design panel members, two experimenters running the study sessions, 100 data collection sessions, one research auditor, and one IT auditor. For the multi-site study, we calculated with 30 consensus design panel members, 30 experimenters, 1000 data collection sessions, two research auditors, and one IT auditor.

research tool	benefits	cost in work hours
		single-site project			multi-site project
		assistant	researcher	programmer	assistant	researcher	programmer
consensus design	protection against post hoc criticism, robustness of the scientific product, increases acceptability among stakeholders	160	224	0	320	560	0
direct data deposition	demonstrates data integrity (with or without data sharing)	2	1	4	2	1	4
born-open data	transparency during the data collection process, earlier reusability, demonstrates data integrity	3	3	8	3	5	6
real-time research report	easy monotoring of study progress, transparency during the data collection process for a larger audience, demonstrates data integrity and consistency of conclusions over time	2	8	8	2	8	8
laboratory logs	easy monitoring of study progress, assess fidelity of protocol delivery, provide meta-data on the session level	8	7	10	34	7	10
manual for experimenters	increased fidelity of protocol delivery	6	20	0	90	20	0
checklist for experimenters	increased fidelity of protocol delivery	1.5	3	0	16.5	3	0
training verified by video recording	demonstrates fidelity of protocol delivery, can aid reproducibility	2	8	0	45	26	0
external research audit	assesses and demonstrates integrity of data collection and study procedures	4	22	19	4	40	19
preregistration	transparency about research and analysis plan	4	48	0	4	48	0
open materials	transparency about research materials, increased reproducibility and reusability	4	8	6	4	8	6
tamper-evident software	demonstrates fidelity of protocol delivery	0	0	1	0	0	1