Table 1.
Non-comprehensive list of important and current challenge efforts and platforms
| Challenge | Scope | Assessment type | Organizers | Website |
|---|---|---|---|---|
| Assemblathon1&2 | Sequence assembly | Objective scoring | UC Davis Genome Center | http://assemblathon.org/ |
| CAFA | Protein function prediction | Objective scoring | Community collaboration | http://biofunctionprediction.org/node/8 |
| CAGI | Systems biology | Objective scoring | UC Berkley/University of Maryland | http://genomeinterpretation.org/ |
| CAPRI | Protein docking | Objective scoring | Community collaboration | http://www.ebi.ac.uk/msd-srv/capri/ |
| CASP | Structure prediction | Objective scoring | Community collaboration | http://predictioncenter.org/ |
| ChaLearn | Machine learning | Objective scoring | ChaLearn Organization (non-for profit) | http://www.chalearn.org/ |
| CLARITY | Clinical genome interpretation | Objective scoring and evaluation by judges | Boston Children’s Hospital | http://www.childrenshospital.org/research-and-innovation/research-initiatives/clarity-challenge |
| DREAM | Network inference and systems biology | Objective scoring | Community collaboration & Sage Bionetworks | https://www.synapse.org/#!Challenges:DREAM |
| FlowCAP | Flow cytometry analysis | Objective scoring | Community collaboration | http://flowcap.flowsite.org/ |
| IGCG-TCGA DREAM Somatic Mutation Calling | Sequence analysis | Objective evaluation | Community collaboration & Sage Bionetworks | https://www.synapse.org/#!Synapse:syn312572 |
| IMPROVER | Systems biology | Objective evaluation and crowd-verification | Phillip Morris International | https://sbvimprover.com/ |
| Innocentive | Topics in various industries | Objective scoring and evaluation by judges | Commercial platform | http://www.innocentive.com/ |
| Kaggle | Topics in various industries | Objective scoring and evaluation by judges | Commercial platform | http://www.kaggle.com/ |
| RGASP | RNA-seq analyses | Objective scoring | European Bioinformatics Institute | http://www.gencodegenes.org/rgasp/ |
| Sequence Squeeze | Sequence compression | Objective scoring and evaluation by judges | Pistoia Alliance | http://sequencesqueeze.org/ |
| X-Prize | Technology | Evaluation by judges | X-Prize Organization (non-for-profit) | http://www.xprize.org/ |
The challenges were chosen based on relevance to cancer genomics or the representativeness of a type of challenge. Different challenges specialize in specific areas of research (see ‘Scope’), and may use different assessment types such as objective scoring against a gold standard, evaluation by judges, or community consensus (‘crowd-verification’). Organizers can be researchers from specific institutions (such as universities or hospitals), a group of diverse researchers from academia and industry collaborating in the challenge organization (community collaboration), not-for-profit associations, or commercial platforms that run challenges as their business model (such as Innocentive and Kaggle). Initiatives such as CAFA, CAGI, CAPRI, CASP, ChaLearn, DREAM, FlowCAP and IMPROVER organize several challenges each year, and only the generic project is listed in this table, with the exception of DREAM, for which we also show the IGCG-TCGA DREAM Somatic Mutation Calling Challenge because of its relevance to this paper. More information about these efforts can be found on the listed websites.