Table 1. System subjective and objective evaluation metrics.

Parent CDS system development
Qualitative	Quantitative
Outcome measure Think-aloud protocol:  Parent review of prototype with interviewer  Parent thinks aloud, communicating understanding of interface, task, and choice selection; Interviewer probes while not guiding Questionnaire:  TAM: validated measure for usability and utility of CDS tools Semi-structured interview:  Questions regarding system design	Outcome measure Error rate: percentage of errors committed by participant, per scenario
Analysis Think-aloud protocol:  Sessions recorded, reviewed, and debriefed by study team members  Most serious usability problems identified and discussed Questionnaire:  Parent rates prototype's usability, functionality, and general satisfaction Semi-structured interview:  Identify missing components or concerns Content reviewed to inform the next iteration of prototype	Analysis Error rate: measure where participant deviates from expected actions in performing task, recorded as either critical error or noncritical error
Pediatric clinician CDS system development
Qualitative
Outcome measure Think-aloud protocol:  Clinician review of prototype with interviewer; prototypes addressed multiple scenarios of parent smoker status, including only one parent smokes, parent and other family member smokes, only other family members smoke, and scenarios where parent may change their mind about accepting treatment after a discussion with the clinician  Clinicians think aloud, communicating understanding of interface, task, and choice selection; Interviewer probes while not guiding Questionnaire:  TAM: validated measure for usability and utility of CDS tools Semi-structured interview:  Questions regarding system design
Analysis Think-aloud protocol:  Sessions recorded, reviewed, and debriefed by study team members  Most serious usability problems identified and discussed Questionnaire:  Clinician rates prototype's usability, functionality, and general satisfaction Semi-structured interview:  Identify missing components or concerns Content reviewed to inform the next iteration of prototype
Clinical staff workflow analysis
Qualitative
Outcome measure Semistructured interview:  Questions describe pros and cons of each workflow at their practice Questionnaire:  Formal evaluated mock-ups of proposed workflow
Analysis Semistructured interview:  Staff asked to describe pros and cons of each workflow at their practice. Questionnaire:  Staff rate suitability for their clinical site and whether the workflow was efficient and easy to manage

Abbreviations: CDS, clinical decision support; TAM, Technology acceptance model.