Table 1.
Additional information for the eight community-based efforts described in the paper. The last row describes other efforts not discussed in the main text
| Name | Domain and Regularity | Website |
|---|---|---|
| KDD Cup | Knowledge discovery and machine learning in various domains. | http://www.sigkdd.org |
| Knowledge Discovery and Data Mining. Every year since launch in 1997. | ||
| InnoCentive | The name mixes Innovation and Incentive. | http://www.innocentive.com/ |
| Crowd-sourcing for problems of commercial interest. | ||
| Founded in 2001. New challenges are released on a rolling schedule. | ||
| Netflix Prize | The name comes from the sponsoring company, Netflix. | http://www.netflixprize.com//index |
| Prediction of user ratings for films, based on previous ratings. | ||
| Only challenge so far, released in 2006, lasted 3 years to complete. | ||
| CASP | Critical Assessment of Techniques for Protein Structure Prediction. | http://predictioncenter.org/ |
| Protein 3D structure prediction assessment. | ||
| Every 2 years since 1994. | ||
| CAPRI | Critical Assessment of PRedicted Interactions. Assessment of predictions of | http://www.ebi.ac.uk/msd-srv/capri |
| protein–protein docking or protein–DNA interaction from 3D structure. | ||
| Goes by Round 22 since 2001. Starts whenever an experimentalist offers an adequate target. | ||
| Predicted structures are submitted 6–8 weeks later. | ||
| DREAM | Dialogue for Reverse Engineering Assessments and Methods. | http://www.the-dream-project.org/ |
| Assessment of quantitative modeling in systems biology. | ||
| Every year since 2006. | ||
| BioCreAtIve | Assessment of Information Extraction Systems in Biology. Evaluating text mining | http://www.biocreative.org |
| and information extraction systems applied to the biological literature. | http://biocreative.sourceforge.net | |
| Every 2 years beginning in 2004. | ||
| FlowCAP | Flow Cytometry Critical Assessment of Population Id Methods. | http://flowcap.flowsite.org/ |
| Evaluation of automated analysis of flow cytometry data. | http://groups.google.com/group/flowcap | |
| Only one iteration on 2010, second one on planning phase. |
Others efforts TunedIT: http://tunedit.org/, RGASP-RNAseq Genome Annotation Assessment Project: www.sanger.ac.uk/PostGenomics/encode/RGASP.html Pittsburgh brain competition: http://pbc.lrdc.pitt.edu/ CAMDA Critical Assessment of Microarray Data Analysis: http://camda.bioinfo.cipf.es/camda2011/ Genome Access Workshop evaluation of statistical genetics approaches: http://www.gaworkshop.