Table 3.

Virtual Vignette Performance Rating Criteria

Performance Measure	Scoring	Description
Engagement	Very Low	Participant doesn't seem to enjoy doing the vignettes, doesn't take ECA seriously. Seems to only be responding to ECA because he has to. May tell the experimenter, “I can't talk to a computer.”
	Low	Participant is doing the vignettes, but not getting into them.
	Moderate	Participant doesn't seem to like/dislike doing the vignettes.
	High	Participant seems to like the vignettes and is engaged. May tell the experimenter he likes the vignettes.
	Very High	Participant is extremely engaged in the vignette activity. May tell the experimenter the vignettes were awesome and that he wishes they were longer.
Verbalizations	Very Low	Two responses -- Participant responds by saying only yes/no.
	Low	Three responses -- Participant says only a little more than yes/no.
	Moderate	Four short responses -- Participant doesn't go into too much detail when responding to ECA.
	High	Five responses -- Participant may have said yes or no, but also provided ECA with detailed information (e.g., questions, preferences, excuses).
	Very High	Six responses -- Participant holds a conversation with ECA, providing him with information about preferences, lots of negotiation, perhaps even provocative statements. This participant never limits his responses to a simple yes/no and stands out from other participants because of how much he says.
Comprehension	Very Low	Has to ask four or more times for experimenter to repeat what ECA said or asks to restart the vignettes because he didn't understand ECA.
	Low	May ask three times for the experimenter to repeat what ECA said.
	Moderate	May ask a couple of times for the experimenter to repeat what ECA said.
	High	Doesn't ask for experimenter to repeat what ECA said but may hesitate once or twice when responding to ECA in order to understand him.
	Very High	No problems understanding what ECA said.
Reaction Time	Very Slow	Participant waits long enough that ECA may say multiple times, “Sorry, I didn't catch that man.” Participant may be told multiple times by the experimenter to respond to ECA.
	Slow	Participant waits long enough that ECA may say once, “Sorry, I didn't catch that man.”
	Moderate	Participant hesitates for about five seconds and may be told once to respond to ECA. Participant seems to think about what he is going to say before responding.
	Fast	Participant responds to ECA within a couple of seconds.
	Very Fast	Participant responds to ECA immediately and sometimes before ECA can finish his sentence.
Emotional Control	Very Low	Not calm at all, very emotional.
	Low	Not too calm but was able to control emotions to a small degree.
	Moderate	Somewhat calm/emotional state went both ways.
	High	Calm/not too emotional.
	Very High	Very calm/not emotional at all.
Information seeking	Very Low	Participant did not ask ECA any questions.
	Low	One question asked.
	Moderate	Two questions asked.
	High	Three questions asked.
	Very High	Four or more questions asked.
Providing information about own preferences	Very Low	Study participant says only “yes” or “no” to the ECA.
	Low	Participant provides one reason why he doesn't wish to go along with ECA.
	Moderate	Participant provides two reasons why he doesn't wish to go along with ECA.
	High	Participant provides three reasons why he doesn't wish to go along with ECA.
	Very High	Participant provides four or more reasons why he doesn't wish to go along with ECA.
Negotiation	Very Low	Study participants does not use any negotiation or bargaining statements (e.g., asking ECA what can be done to avoid a physical fight).
	Low	One negotiation or bargaining statement.
	Moderate	Two negotiation or bargaining statements.
	High	Three negotiation or bargaining statements.
	Very High	Four or more negotiation or bargaining statements.
Being non-provocative	Very Low	Four or more provocative or oppositional statements by study participant and/or study participant curses at the ECA multiple times.
	Low	Three provocative/oppositional statements and/or study participant may curse at ECA one time.
	Moderate	One or two provocative/oppositional statements and study participant doesn't curse at ECA.
	High	No provocative/oppositional statements, but study participant's tone could be more calm.
	Very High	No provocative/oppositional statements and study participant is not provocative in any way.