Table 2.

Some advantages and limitations of challenge-based methods assessment, along with barriers to participation in them

Advantages	Limitations	Participation barriers
Reduction of over-fitting	Narrower scope compared to traditional open-ended research	Incentives not strong enough to promote participation
Benchmarking individual methods	Ground truth needed for objective scoring	No funding available to support time spent participating in challenges
Impartial comparison across methods using same datasets	Mostly limited to computational approaches	Fatigue resulting from many ongoing challenges
Fostering collaborative work, including code sharing	Requires data producers to share their data before publication	Time assigned by organizers to solve a difficult challenge question may be too short
Acceleration of research	Sufficient amount of high-quality data needed for meaningful results	Lack of computing capabilities
Enhancing data access and impact	Large number of participants not always available	New data modality or datasets that are too complex or too big poses entry barrier
Determination of problem solvability	Challenge questions may not be solvable with data at hand	Challenge questions not interesting or impactful enough
Tapping the ‘Wisdom of Crowds’	Traditional grant mechanisms not adequate to fund challenge efforts	Cumbersome approvals to acquire sensitive datasets
Objective assessment	Difficulties to distribute datasets with sensitive information
Standardizes experimental design