Table 1.

Summary of recommended requirements for a vignette-based online symptom checker (OSC) clinical evaluation standard.

Category	Requirements
Vignettes	Illustrate the method for determining the minimum number of vignettes required for OSC evaluation. Specify the minimum information to be included in each vignette. Provide guidance on determining the conditions, symptoms, or spread of cases to be included and assurance that this appropriately represents the target user population. Specify vignette origin requirements (eg, simulated vs real-world cases).
Clinician assessment	Specify the appropriate clinicians (including role, speciality, and seniority) to be used in the assignment of ‘gold standard’ labels. Illustrate the method used for compilation of a gold standard (eg, averaged single blinded assessment or consensus discussion).
Triage	Specify standardized triage categories, including setting and time periods. Provide guidance on the use of triage gold standards as a range for both urgency and setting.
Differential diagnosis	Illustrate the method for comparing OSC differential diagnosis list to gold standard dispositions.
Accuracy	Illustrate the accuracy and safety score calculation method, accounting for outcomes that fall both below and above the gold standard. Specify the minimum accepted accuracy and safety scores.
Safety netting	Specify how safety netting contributes to product safety scores.
Inputters	Specify the minimum interrater reliability scores for inputters of vignettes. Illustrate the method for determining the number of tests and inputters required for each vignette. Specify the appropriate characteristics for vignette inputters (eg, medical education level and affiliation with developers).