Table 3. Popular online research platforms, their main features, strengths and weaknesses, as reported by their developers.

Survey conducted through Google Forms, on 13-3-2015, which is not listed in the below table on account of being mostly questionnaire-focused and thus ‘neutral territory’ for responders.

		JsPsych	Inquisit	LimeSurvey	ScriptingRT	Qualtrics	SoPHIE	Tatool	Unipark	WebDMDX	Xperiment
Open-source		yes	no	yes	yes	no	citewarea	yes	no	no	yes (in beta)
Yearly Fee for one researcher (USD)			1495			?a	?b		138.12
Publish directly to crowd sourcing sitesc		nod	with addonse	no	no	MTurk	yes	yesf	no	nog	MTurk, ProlificAc
Questionnaire vs Perceptual Research focus (Q vs R)
Coding required for	Software setup	yes	no	no	yes	no	no	no	no	no	yes
	Creating a study	yes	script based	no	script based	no	no	no	no	script based	script based
Possible trial orderings	Random	yes	yes	yes	yes	yes	yes	yes	yes	yes	yes
	Counterbalanced	yes	yes	no	no	yes	yes	yes	yes	yes	yes
	Blocked	yes	yes	yes	no	yes	yes	yes	yes	yes	yes
Reaction times measurable in*		ms	msh	msi	ms	ms	ms	ms	ms	msh	ms
Image, sound & video stimuli*		yes	yes	yes	yes	yes	yes	yes	yes	yes	yes

Notes.

^*Suffixed rows requested by reviewer/data not provided by developers.

^aMany academic institutions have licenses with Qualtrics already. Individual academic pricing was not disclosed to us and could not be found via search engines. Note also that some features (e.g., De Leeuw (2014) more advanced randomization) may require a more expensive package.

^b“Free as CiteWare, Commercial Hosting Service from SoPHIE Labs (950 USD/year)”.

^cAlthough all platforms let the researcher provide a URL where the participant can undertake a study, some crowd-sourcing sites need to communicate directly with the testing software in order to know, for example, whether the participant should be paid.

^d“None directly; but it can be used to publish on any platform that allows for custom JavaScript and HTML content”.

^eSee http://www.millisecond.com/support/docs/v4/html/howto/interopsurveys.htm.

^f“Any crowd sourcing site that allows an external link to Tatool to run an experiment (no login required)”.

^g“It uses an HTML POST command so pretty much anything, depends how skilled you are. We provide a site running a general purpose script to gather data and email it to experimenters should people not be in a position to setup a site to gather the data”.

^hAs discussed in the text, both these packages run outside of the browser and thus likely to more reliably and more accurately measure reaction time.

ⁱThis was a topic of contention amongst our reviewers. However, as LimeSurvey is extendable with packages such as http://www.w3.org/TR/hr-time/, timing accuracy within this framework is quite on par with other browser based frameworks.