Table 1.

Outcome measures.

Outcome indicator		Goal	Analytic approach
Impact on decision-making
	Decision consistency	To explore decision consistency between the users and the CDSa model to research whether having the CDS model increases uniformity among the decisions taken compared to decisions without the model.	Comparing the taken decisions: Between the CDS model output and the test group to assess the extent of how much the users agreed with the model output. Between test and control groups to assess similarity in decisions taken with and without the model output. Potential decisions: To discharge, with x, y, and z treatment. Move to another department (which?). Take additional tests for an extended investigation. To send a referral or follow-up at outpatient clinical, primary care, or home care. To stay for some more days.
	Decision quality	To explore whether the CDS model could improve decisions.	Comparing decisions taken by the test group with: Historical data—what decision was actually taken for the patient and what event followed afterwards. Control group’s decisions on the same patients to control the influence of the experiment and set up over the decisions taken: do decisions differ from the test group and the historical data?
	Work efficiency	To explore performance changes due to using the CDS model output.	Measuring speed of decision-making for both test and control groups. Comparing the average speed between groups with and without CDS.
Impact on experience with decision-making
	Perceived benefits	Clinicians’ attitude toward perceived benefit for patients and clinicians of AIb.	Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q1).
	Knowledge	Knowledge sufficiency and possible gaps.	Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q5).
	Confidence	Confidence in making decisions using the algorithm (trust in the algorithm and data, and self-confidence).	Asking the participants in the test and control groups to take decisions for some patients with and without the CDS model output. Clinicians will be asked to rate their decision confidence and indicate to what extent the CDS output helped them using a 5-point Likert scale (1=not at all, 5=a great deal) [16]. Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q3 and Q7).
	Reliability and validity	How reliable and valid are the suggestions by the algorithm—perception?	Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q3).
	Perceived service quality	How is the perception of the overall clinician-provided service perceived?	Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q2 and Q6).
	Unintended consequences	Unintended consequences are foreseen.	Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q4).
	Intention of use	Obtaining an indication of worthiness to continue developing the AI-based system.	Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q1, Q2, and Q6).
Implementation aspects
	Workflow integration	How integrable is the solution into the current workflows?	Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q4 and Q8).
Usability
	Perceived ease of use	Perception of the features, human-computer interface.	Interview with nurses and physicians who were exposed to the CDS model output (Multimedia Appendix 1, Q5).

^aCDS: clinical decision support.

^bAI: artificial intelligence.