Table 4.

Reader metrics

Study	Reference standard	Number of reference standard readers	Index test readers and profession	Index test reader attributes	Index test instrument
Abboud et al17	NA	NA	Five radiologists	Two chest fellowship, two fellowship trainees and one resident	Binary (positive/negative)
McNulty et al9	Radiologist working in consensus	Two	13 radiologists	12 board-certified neuroradiology experts, 1 board-certified spinal and musculoskeletal expert	Six-point confidence scale
Tewes et al15	NA	NA	Three radiologists	Three radiologists with 3, 4 and 6 years' experience, respectively	Five-point confidence scale
Panughpath et al10	Discordant studies assessed by fellowship-trained neuroradiologist	Two	Two radiologists	Not reported	Binary (positive/negative) plus type categories, e.g. extradural, subdural, subarachnoid, intraparenchymal or intraventribular
Johnson et al18	Cases reported as clinically positive were further reviewed	Three (one initial clinical radiologist plus two additional)	Two radiologists	Two fellowship trained but junior radiologists	Binary (positive/negative)
Yoshimura et al19	Radiologists working in consensus plus accuracy of report confirmed by MRI and clinical records	Two	Nine radiologists	Six general radiologists and three neuroradiologists with 3–17 years' experience	Continuous confidence scale
Park et al8	Neuroradiologist	One	Five emergency department physicians	Three attending and two senior residents	Five-point confidence scale
John et al2	Clinical report of non-study senior radiologist	One	Three radiologists	Three attending with at least 10 years' experience each	Descriptive report. Non-study radiologist classified discrepant diagnosis as major or minor
McLaughlin et al1	Clinical report of radiologist	One	Two radiologists	Two radiologists with 5 and 16 years' experience	Descriptive report. Discrepancies classified according to American College of Radiologist's RadPeer classification system
Bhatia et al20	NA	NA	Five radiologists	One board-certified neuroradiologist, three fourth-year radiology residents and one second-year radiology resident	Binary (positive/negative) plus type categories, e.g. disc herniation and/or reason category, e.g. gradient echo signal abnormality
Lee et al16	Results of IVU study (as opposed to plain AXR) and clinical records	One	Two radiologists	Not reported	Five-point confidence scale

AXR, abdominal radiograph; IVU, intravenous urography; NA, not applicable.