Table 1.

Potential problems with decision‐making scenarios and likely consequences.

Problem	Description	Likely consequence
Implausibility	Scenario is unlikely to occur in real life or not relevant for some or all participants (e.g., a decision about managing a pregnancy for a mixed‐sex sample).	Low engagement among participants, high rates of missing data or ‘I am not sure’ responses, and overall poor ecological validity.
Incomprehensibility	Scenario is difficult to understand due to the presence of medical jargon, inappropriate language, or description of situations that that are difficult for participants to visualise in practical terms.	High rates of missing or ‘unsure’ responses, particularly among participants with lower education or lower health literacy.
Monotonicity	Scenario is presented in a way that strongly favours a particular response.	Ceiling effects on the most popular response, floor effects on the least popular response/s, and a tendency towards very low decisional conflict scores. Low between‐participant variability, resulting in poor discrimination and poor sensitivity to change in response to interventions.
Ambiguity	Scenario presents an ‘impossible’ choice that would likely be difficult for any participant, regardless of personal experience and/or the presence of decision support.	High rates of ‘I am not sure’ responses, platykurtic distribution of decisional conflict scores and poor sensitivity to change in response to interventions.