Table 1. Possible outcomes: a non-exhaustive list.
| Outcome | Issues |
|---|---|
| Benefit outcomes | |
| Fairness | Lack of a relatively objective criterion (gold standard) measure (
Brezis & Birukou, 2020). Operationalised as
distributive fairness (fairness of outcomes), may require large numbers/long timescales, to spot differences in clustering of grants. Procedural / informational fairness would require researchers to observe and code committee work or an adjudication committee to assess content of rejection letters. |
| Efficiency: time to
deliberation |
Objective, continuous (therefore efficient) measure and has been used successfully (
Bendiscioli
et al., 2022)] but
may not be seen as important to the public. |
| Efficiency: appeals | Objective, dichotomous measure. May require large sample sizes, depending on base rate. Not universally
applicable and not every funder permits appeals. |
| Diversity | Requires operationalisation by applicant demographic (gender, ethnicity, etc.) or topic (academic disciplines and
research methodologies). The latter might require coding manuals and coders / adjudication committees to resolve. |
| High-risk, high
reward projects |
Risk is subjective and would require researchers to observe and code applications or an adjudication committee.
Reward would require long timescales |
| Exceptional
scientific advances |
Requires a long timescale and large numbers (very rare event). Would require researchers to observe and code
applications or an adjudication committee running over years. |
| Harm outcomes | |
| Application quality | Requires coding manuals and coders / adjudication committees to resolve. |
| Questionable
research practices |
Subjective. Would require advertisement of the RCT between lottery and usual practice, as well as two researchers
to code grant applications against a framework for QRP. |
| Reputational
damage to funder |
Not subject to experimental design. Can perhaps be operationalised via perceptions of individual scheme and/or
individual scheme applicants. |
| Stigmatisation of
awardee |
Timescales likely to be undesirable. Measurement likely to be problematic. |