Table 2.

Q-methodology results

Most important criteria	A: Early development	B: Early development	C: Late phase III	D: Post-marketing
A typical survey can be conducted at relatively low costs	✓	✓
Data can be collected during quick sessions with participants		✓	✓
Low frequency of sessions required by patients	✓	✓	✓
Relatively quick delivery of preparation, data collection, and analysis	✓	✓	✓	✓
A large number of attributes can be explored				✓
Suitable to study preferences in a small sample size	✓	✓		✓
A low cognitive load on patients	✓	✓	✓	✓
Does not need an education tool or preparatory instructions in order to enhance participant comprehension		✓	✓
Publically acknowledged by your organisation’s guidelines as an acceptable method to study preferences			✓	✓
New attributes can be added without making prior results invalid	✓	✓	✓
Can be used to collect data from more than one participant in a single session			✓
The analysis can calculate risk attitudes, like risk tolerance, and calculate how value functions bend due to the presence of uncertainty in the participant	✓	✓	✓	✓
Explores the reasons behind a preference in detail	✓	✓	✓	✓
Can estimate weights for attributes	✓	✓	✓	✓
Estimates trade-offs that patients are willing to make among attributes	✓	✓	✓	✓
Can quantify heterogeneity in preferences	✓	✓	✓	✓
Internal validity can be established	✓	✓	✓	✓
External validity can be established	✓	✓	✓	✓
Outcomes can refer to a course of health over time (as opposed to a constant health state)			✘	✘
Sensitivity analysis is possible	✘	✘	✘	✘
Can combine quantitative and qualitative methods		✘	✘	✘
Applies validation tests	✘		✘	✘
Results can be reproduced by an (independent) researcher for reproducibility	✘	✘	✘	✘
Applies tests for consistency			✘	✘
Can be conducted without the need for specialized software (beyond Excel)
Can be conducted without programming skills
Researcher does not need to supervise the data collection
Does not require hypothetical scenarios
Attributes and attribute levels can be determined as part of the method itself (internal identification)
Data saturation can be achieved relatively quickly
Does not require model estimations
Outcomes can be expressed in a particular format (e.g. probability scores, marginal rates of substitution, monetary values)
Outcomes can refer to a constant health state (as opposed to a course of health over time)
Uses respondent validation by asking participants to check their data or responses
Validates through triangulation

✓ Criteria considered important in the Q-methodology, included in the AHP

✘ Criteria considered important in the Q-methodology, but not included in the AHP for the following reasons: 1. The criterion does not sufficiently discriminate between each method (i.e. every method would perform the same way under the criterion), 2. The criterion reflects an element of good study conduct, and not a unique aspect of a method itself, 3. The criterion could be absorbed into other similar criteria, in order to avoid the oversaturation of themes