Table 1.

Advantages and limitations of various techniques used to prepare gold standard data

Technique	Advantages	Limitations
Trusted technology	High accuracy Direct, usually, no computational inference is required	Carries high cost Does not scale
Alternative technology	Direct, usually, no computational inference is required	Not necessarily more accurate
Multiple ordinary technologies	Using a consensus between the technologies allow reducing the number of false positives compared with each individual technology	Disagreement between used technologies results in the incompleteness of the gold standard
Mock community	Ground truth is fully known, because raw data are generated from prepared gold standard	The small number of items (e.g., microbial species) compared with reality The designed community is artificial
Expert manual evaluation	Most suitable for specialist understanding	Does not scale Lack of formal procedure, limiting comparison of results produced by different experts
Curated database	Allows access to sensitivity, by comparing the number of elements in the sample and the database	Incompleteness of curated databases results in limited ability to define true positives and false negatives
Curated software input	Ground truth is fully known, because raw data are generated from prepared gold standard	Does not validate on real inputs, which usually contain errors
Computational simulation	Ground truth is fully known, because raw data are generated from prepared gold standard Cost-free generation of multiple gold standards	Technology is simulated, and cannot capture true experimental variability and will always be less complex than real data Gold standard data are artificial