Table 6.

Reference Standard Levels

Level	Description
I	A reference standard that is either a prognostic standard, a clinical outcome, or a biomarker standard. If a prognostic standard, it is determined by an independent reading center. If either a prognostic standard or a biomarker, it is validated against clinical outcome, and temporal drift, reproducibility, and repeatability metrics are published.
II	A reference standard established by an independent reading center. Temporal drift, reproducibility, and repeatability metrics are published. A level II reference standard has not been validated against clinical outcome or a prognostic standard.
III	A reference standard created from the same method as used by the AI, by adjudicating or voting of multiple independent expert readers. The readers are documented to be masked, and reproducibility and repeatability metrics are published. A level III reference standard has not been validated against clinical outcome or a prognostic standard and does not have known temporal drift, reproducbility or repeatability.
IV	All other reference standards, created by single readers or nonexpert readers, possibly without an established protocol. A level IV reference standard has not been validated against clinical outcome or a prognostic standard and does not have known temporal drift, reproducibility, or repeatability metrics, and the readers may not have been masked.

AI = artificial intelligence.