Table 10.

Limitations of Current Grant Review Models

Review Mechanism	Presence of Implicit Bias	Presence of Strategic Voting (based on connection to PI)	Tie Breaking Mechanism	Influence of Outlier Ratings	Sources of Reviewer Input into Recommended Funding Order	Recommended Funding Order Directly or Indirectly Determined	Level of Uncertainty in Recommended Funding Order
Unblinded Panel Review Rating of Proposal Sets	Direct	Direct	None	Medium	Rating Alone	Indirectly (Summary stats of Ratings)	None
Blinded Panel Review Rating of Proposal Sets	Indirect	Indirect	None	Medium	Rating Alone	Indirectly (Summary stats of Ratings)	None
Rating Review of Individual Projects	Direct	Direct but Only Impacting a Few Proposals	None	High	Rating Alone	Indirectly (Summary stats of Ratings)	None
Partial Lottery with Initial Rating of Proposal Sets	Initially Direct and then Random Selection	Initially Direct and then Random Selection	Initially None and then Random	High (Initial Rating with no Panel)	Initial Rating and then Random Selection	Initially Indirectly (Summary stats of Ratings) and then Randomly	Initially None and then Random
Unblinded Panel Review Rating/Ranking of Proposal Sets using Mallows-Binomial Model	Direct	Direct	Integrated Score	Low	Both Rating and Ranking	Directly	Estimated Confidence Intervals