Table 2.

Case study comparison of usability evaluation features.

Feature and usability type		Case study 1 (Integrated Clinical Prediction Rule 2)	Case study 2 (Avoiding Diabetes Thru Action Plan Targeting)
Objectives
	Academic	To generate evidence on the optimal adaptation of clinical decision-support tools	To generate evidence on the clinical impact of an electronic health record-enabled prediabetes counseling tool
	Pragmatic	Tool adaptation and identification of issues in tool build before widespread deployment	User feedback for recommendations to tool developers
Methods used: Academic and pragmatic		Direct observation Think-aloud Near-live Live testing Semistructured group interview (postdeployment)	Direct observation Think-aloud Near-live Live testing
Setting: Academic and pragmatic		Laboratory and in situ	Laboratory and in situ
Core team: Academic and pragmatic		9 members (expertise: primary care, clinical decision support, informatics, electronic health records, usability, qualitative research, and graphic design)	6 members (expertise: primary care, health psychology, diabetes education, nutrition, informatics, usability, and graphic design)
Number of participants: Academic and pragmatic		Think-aloud=12 clinicians Near-live=12 clinicians (same) Live=3 clinicians and 6 encounters Postdeployment=75 clinicians and 14 sites (group interviews)	Think-aloud=7 clinicians Near-live=6 clinicians
Data capture: Academic and pragmatic		Note taking Audio recording of sessions Video recordings Screen capture	Note taking Audio recording of sessions Screen capture
Termination criteria: Academic and pragmatic		Termination with data saturation for current iteration	Termination with data saturation for current iteration
Data analysis
	Academic	Qualitative thematic analysis by 2 independent coders	Qualitative thematic analysis by 2 independent coders
	Pragmatic	Thematic analysis of observational field notes	Thematic analysis of observational field notes
Output
	Academic	Detailed data tables and results reporting	Detailed data tables and results reporting
	Pragmatic	Summary reports from field notes	Summary reports from field notes
Dissemination
	Academic	Publication of protocol and usability findings from think-aloud, near-live, and live testing in peer-reviewed journals	Publication of protocol and usability findings from think-aloud and near-live testing in peer-reviewed journals
	Pragmatic	Research team Electronic health record development team	Research team Electronic health record development team
Time frame
	Academic	Think-aloud or near-live usability 16 months from the beginning of data capture to the publication of findings	Think-aloud or near-live usability 11 months from the beginning of data capture to the publication of findings
	Pragmatic	Think-aloud or near-live usability 2 months from the beginning of each phase of data capture to the completion of all summary reports	Think-aloud or near-live usability 1 months from the beginning of each phase of data capture to the completion of all summary reports