Table 1.
Methodological differences in conducting interactive experiments in the laboratory and on MTurk
| Phase/challenge | Laboratory | Online (MTurk) |
|---|---|---|
| Recruitment | ||
| Show-up fees | Typically a small part of total payoffs. Guaranteed when participant shows up to the session | Relatively large show-up fees promote recruitment rates, thereby facilitating prompt group formation. Experimenter can approve or reject the task submitted; if rejected no fee is paid |
| Inviting participants | Invitations sent well in advance, participants commit to a session. Recruitment often from a pre-existing database | Sessions advertised online as HITs and can be completed immediately |
| Selection into the experiment | At sign-up, participants know very little about the experiment. Details of the task are communicated once participants are in the laboratory | Experiments are typically advertised as HITs with a brief task description. ‘Workers’ browse available HITs and accept those of their preference |
| Experienced participants | Invitation conditioned on well-defined criteria of the laboratory’s records | HITs targeted at subsets of MTurk workers; experimenter can specify exclusion criteria. Many MTurk workers will have participated in many prior studies |
| Session start-up | ||
| Duplicate participants | Registration protocols usually prevent duplicate participation | Amazon acts against multiple worker accounts, but they exist |
| Comprehension | Participants can ask questions; comprehension questions ensure understanding | Experimenter is physically absent and cannot answer questions directly. Compulsory comprehension questions can be added but may make experiment (too) long for some participants |
| Experimental interactions | ||
| Forming groups | Easy to guess how many participants will attend; group settings can be pre-defined | Hard to guess how many participants will attend; groups can be constructed ‘on the fly’ |
| Deception | In experimental economics deception is prohibited and laboratories foster reputations for non-deception | Because all requesters use the same subject pool, some participants may have experienced deception because requesters from other disciplines may use it |
| Communication | Hardly an issue; experimenter can restrict communication between subjects | Participants may in principle collude through external channels though this is difficult in practice |
| Experimental flow | Closed form software like z-Tree specifies session progress | Scripted browser navigation specifies progress |
| Attrition (‘dropout’) | Hardly an issue; participants that start a session usually finish it | Major challenge to internal validity, if dropout rates vary with treatment, selection bias may arise |
| Payment | ||
| Payments | Cash usually paid upon completion | Automatic transfer through Amazon |
| Cost per participant | Relatively high but predictable | Relatively low but varies with attrition |